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SUMMARY 

Researchers  interested  in  studying  how  teams  collaborate  have  been  trying  for  a  number  of  years 
to  develop  an  empirically  based  model  of  team  collaboration.  The  goal  of  this  research  was  to 
validate  the  existence  and  facilitate  a  better  understanding  of  the  stages  and  underlying 
macrocognitive  processes  of  team  collaboration.  Communication  transcripts  previously  collected 
during  a  team  collaboration  experiment  using  a  Noncombatant  Evacuation  Operation  scenario 
were  used  for  the  present  research  endeavor.  The  transcripts  were  recoded  for  the  underlying 
macrocognitive  major  processes  and  subprocesses  of  team  collaboration  using  the  newest 
taxonomy  and  the  accompanying  revised  set  of  definitions,  and  then  they  were  re-analyzed. 

The  results  of  the  re-analyses  support  the  contribution  of  the  four  collaboration  stages  during  the 
process  of  team  collaboration,  with  the  caveat  that  their  consistency  appears  to  be  domain 
dependent.  The  contribution  of  all  five  major  macrocognitive  processes  during  team 
collaboration  was  also  significant;  however,  whether  or  not  this  result  is  also  dependent  upon  the 
problem-solving  domain  is  still  unknown.  The  results  of  the  present  re-analyses  also  provided 
support  for  some  of  the  macrocognitive  subprocesses  in  tenns  of  both  their  consistency  and 
contribution.  There  were  also  significant  differences,  depending  on  the  collaboration  mode  used 
by  the  teams,  in  terms  of  the  collaboration  stages,  macrocognitive  major  processes,  and 
macrocognitive  subprocesses.  The  next  phase  in  the  evolution  of  a  conceptual  model  of  team 
collaboration  is  proposed  and  areas  for  future  research  are  also  discussed. 
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INTRODUCTION 

Team  collaboration  and  team  problem  solving,  whether  in  an  educational  environment,  in  a 
work  setting,  or  during  military  operations,  have  been  studied  for  many  years.  However,  the 
“advent  of  global  networking  brings  the  promise  of  greatly  expanded  collaboration  opportunities 
-  both  for  learning  and  for  working  together  without  geographic  limitations”  (Stahl,  2006,  p. 
285).  Team  collaboration  tasks,  such  as  learning  or  problem  solving,  through  a  computer- 
mediated  environment,  such  as  the  World  Wide  Web,  require  the  facilitation  of  communication 
between  team  members  who  may  be  located  separately  geographically.  Team  members  may  also 
need  to  communicate  over  long  periods  of  time,  and  not  necessarily  continually.  It  is  important 
that  the  different  forms  of  communication  that  are  needed  (e.g.,  threaded  discussion,  picture 
sharing,  whiteboard,  etc.),  depending  on  the  task  involved  as  well  as  both  geographic  and 
temporal  difference  considerations  be  supported  by  the  system  (Stahl,  2006).  In  order  to 
determine  the  most  useful  forms  of  communication  for  various  types  of  tasks,  and  to  facilitate 
the  best  possible  designs  for  team  collaboration  tools,  a  good  model  of  team  collaboration  is 
needed. 

Computer-supported  collaboration  tools  have  enormous  potential  to  facilitate  learning, 
communication,  problem  solving,  and  to  improve  our  lives  in  general.  Unfortunately,  the 
software  design  process  that  supports  the  necessary  forms  of  communication  for  various  tasks  is 
a  difficult  one.  One  obstacle  hindering  the  development  process  is  the  lack  of  a  clear 
understanding  of  the  underlying  mechanisms  employed  by  teams  to  collaborate  effectively 
(Stahl,  2006).  A  number  of  researchers  are  studying  this  area  (e.g.,  Cooke,  Kiekel,  Salas,  Stout, 
Bowers,  and  Cannon-Bowers,  2003;  Hutchins,  2008;  Warner,  Biron,  and  Burkman,  2008; 
Warner  and  Wroblewski,  2004,  2005,  2006)  and  through  such  empirical  research,  a  clearer 
understanding  of  team  collaboration  should  evolve. 

Researchers  in  the  field  of  team  collaboration  have  proposed  a  number  of  models  of  team 
collaboration  (e.g.,  Cooke,  2005;  McNeese,  Rentsch,  and  Perusich,  2000;  Orasanu  and  Salas, 
1992;  Stahl,  2000,  2006).  The  Cognitive  Model  of  Team  Collaboration  proposed  by  Warner, 
Letsky,  and  Cowen  (2005),  shown  in  figure  1,  is  based  on  team  collaboration  stages  and 
macrocognitive  processes  and  has  undergone  an  empirical  research  based  evolution  since  its 
early  conceptualization  in  2002  (see  Warner  and  Letsky,  in  press).  Warner,  Letsky,  and  Cowen’s 
(2005)  early  research  used  a  murder  mystery  scenario  and  a  Noncombatant  Evacuation 
Operation  (NEO)  scenario  to  investigate  team  collaboration  and  problem  solving.  The 
researchers  used  the  murder  mystery  for  the  first  experiment  in  order  to  investigate  two 
variables:  1)  face-to-face  versus  asynchronous-distributed  collaboration  mode  and  2) 
homogeneous  versus  heterogeneous  knowledge  distribution.  For  the  second  experiment,  the 
researchers  decided  to  use  an  NEO  scenario  in  order  to  investigate  both  face-to-face  and 
asynchronous-distributed  team  collaboration  further.  During  this  experiment,  the  researchers  also 
examined  static  versus  dynamic  knowledge  uncertainty. 
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Unlike  the  murder  mystery  scenario  that  had  a  fixed  solution,  the  NEO  scenario  created  by 
Warner,  Letsky,  and  Shuck  (2004)  included  numerous  possible  solutions  that  varied  in  terms  of 
how  optimal  they  were  when  compared  to  those  planned  by  experts.  (The  NEO  scenario  will  be 
discussed  more  in  a  later  section.)  The  most  significant  finding  in  terms  of  the  differences 
between  these  two  different  types  of  solutions  was  that  teams  solving  fixed  solution 
collaborative  problems  (e.g.,  the  murder  mystery  scenario)  did  not  go  through  the  Outcome 
Evaluation  and  Revision  stage.  However,  teams  that  collaborated  on  problems  where  they 
needed  to  develop  optimized  solutions  (e.g.,  the  NEO  scenario)  spent  a  significant  amount  time 
in  each  of  the  four  stages  outlined  in  the  model  (Warner,  Letsky,  and  Cowen,  2005). 
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Figure  1 :  Warner,  Letsky,  and  Cowen’s  (2005)  Model  of  Team  Collaboration 

Recently,  a  number  of  team  collaboration  researchers  (Letsky,  Warner,  Fiore,  Rosen,  and 
Salas,  2007)  proposed  some  revisions  to  the  Warner,  Letsky,  and  Cowen  (2005)  model  of  team 
collaboration  and  a  corresponding  new  taxonomy.  The  new  taxonomy  of  team  collaboration  that 
Letsky,  and  colleagues  proposed  is  shown  in  figure  2.  It  includes  all  4  of  the  team  collaboration 
stages  (these  remained  unchanged),  5  new  major  macrocognitive  processes,  and  22 
macrocognitive  subprocesses  (see  Letsky,  et  al.,  2007),  and  is  the  basis  for  the  present  re¬ 
analysis. 


2 


NAWCADPAX/TR-2008/43 


Major  Macrocognitive  Processes 
(Shaded)  and 

Sub-Macrocogitive  Processes 


Individual  Knowledge  Building 


Iterative  Information  Collection 


Individual  Task  Knowledge 
Development 


Individual  Mental  Model 
Development 


Team  Knowledge  Buildin 


Pattern  Recognition  and  Trend 
Analysis 


Team  Mental  Model  Development 


Recognition  of  Expertise 


Sharing  Unique  Knowledge 


Uncertainty  Resolution 


Knowledge  Interoperability 


Developing  Shared  Problem 
Conceptualization 


Visualization  and  representation  of 
meaning 


Building  common  ground 


Knowledge  sharing  and  transfer 


Team  Shared  Understanding 


Team  Consensus 

Development _ 

Critical  thinking 


Mental  simulation 


Intuitive  Decision  Making 


Iterative  Information  Collection 


Solution  Option  Generation 


Storyboarding 


Team  Pattern  Recognition 


Negotiation  of  Solution  Alternatives 


Outcome 

Appraisal 


Feedback  Interpretation 


Replanning _ 

Team  Pattern  Recognition 


Collaboration  Stages 

Knowledge 

Construction 

Team 

Problem 

Solving 

Team 

Consensus 

Evaluation 

and  Revision 

Figure  2:  Letsky  and  Colleagues’  (2007)  Team  Collaboration  Taxonomy  Constructs 
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MACROCOGNITION,  MACROCOGNITIVE  PROCESSES,  AND  THE  NEW  TAXONOMY 
OF  TEAM  COLLABORATION 


“Macrocognition  in  problem  solving  teams  involves  the  study  of  individual  and  collective 
knowledge  building  and  utilization  of  knowledge  in  the  problem  solving  process”  (Letsky,  et  ah, 
2007,  p.  5).  Letsky  and  colleagues  (2007)  also  emphasized  the  importance  of  both  the 
internalized  and  externalized  processes  (processes  that  occur  both  inside  and  outside  the  head, 
respectively)  that  occur  during  team  collaboration,  and  included  them  in  their  latest  definition  of 
macrocognition. 

“Macrocognition  is  defined  as  the  internalized  and  externalized  high-order  cognitive  processes  employed 
by  teams  to  create  new  knowledge  during  complex,  one-of-a-kind,  collaborative  problem  solving.  High- 
order  is  defined  as  the  process  of  combining,  visualizing,  and  aggregating  information  to  resolve  ambiguity 
in  support  of  the  discovery  of  new  knowledge  and  relationships”  (Letsky,  et  al.,  2007,  p.  7). 

In  this  paper,  Letsky  and  colleagues  (2007)  provided  a  revised  taxonomy  and  operational 
definitions  designed  to  facilitate  a  better  understanding  of  team  collaboration  during  complex 
team  problem  solving.  The  researchers  based  the  revisions  on  the  lessons  learned  from  the  two 
empirical  experiments  described  by  Warner,  Letsky,  and  Cowen  (2005).  The  researchers 
proposed  a  set  of  macrocognitive  major  processes  and  nested  subprocesses  that  support  each  of 
the  team  collaboration  stages,  all  of  which  have  been  evolving  since  2002. 

The  four  stages  used  to  help  describe  team  collaboration  are:  1)  Knowledge  Construction,  2) 
Collaborative  Team  Problem  Solving,  3)  Team  Consensus,  and  4)  Outcome,  Evaluation,  and 
Revision  (see  Letsky,  et  al.,  2007;  Warner,  Letsky,  and  Cowen,  2005;  Warner  and  Wroblewski, 
2005).  However,  teams  may  go  through  the  stages  in  various  orders  and  an  unlimited  number  of 
times  for  each  stage.  Therefore,  the  stages  are  not  sequentially  ordered,  but  rather,  are  dynamic 
in  nature. 

For  example,  a  team  may  begin  collaborating  on  a  problem  by  establishing  a  plan  toward 
solving  the  problem.  This  falls  under  the  Collaborative  Team  Problem  Solving  stage.  The  team 
may  then  read,  combine,  and  visualize  some  information  pertinent  to  a  given  problem.  This  falls 
under  the  Knowledge  Construction  stage.  The  team  members  may  then  communicate 
information  about  the  problem  and/or  analyze  data  to  try  to  come  up  with  a  solution.  This  also 
falls  under  the  Collaborative  Team  Problem  Solving  stage.  They  may  then  return  to  reading  and 
combining  more  information  (Knowledge  Construction),  then  begin  to  negotiate  solution  options 
(Team  Consensus),  and  then  work  as  a  team  to  rationalize  part  of  a  possible  solution 
(Collaborative  Team  Problem  Solving).  After  this,  the  team  may  try  to  evaluate  the  solution  they 
chose  against  the  goal  of  the  problem  (Outcome,  Evaluation,  and  Revision).  If  the  team  members 
think  that  they  need  to  revise  any  part  of  the  solution  (also  part  of  the  Outcome,  Evaluation,  and 
Revision  stage),  they  may  first  need  to  read,  combine,  and  visualize  more  infonnation  or  ask 
other  team  members  questions  to  help  clarify  the  information.  Reading  information  or  asking 
other  team  members  clarification  questions  would  both  fall  under  the  Knowledge  Construction 
stage.  The  team  may  also  need  to  use  data  and  information  to  help  justify  or  rationalize  a 
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solution  (both  classified  as  Collaborative  Team  Problem  Solving)  before  reaching  final 
agreement  on  the  solution  (Team  Consensus).  Researchers  are  still  studying  exactly  how  teams 
flow  through  the  stages  and  macrocognitive  processes  (Letsky,  et  ah,  2007).  Warner,  Letsky,  and 
Cowen  (2005)  suggested  that  it  might  depend  on  a  number  of  factors.  For  example,  they  found 
team  collaboration  differences  that  depended  upon  the  type  of  team  collaboration  problem  (e.g., 
fixed  solution  versus  optimum  solution)  and  based  upon  whether  the  teams  communicated  in  an 
asynchronous-distributed  or  in  a  face-to-face  manner.  The  level  of  difficulty  of  the  problem  (e.g., 
the  amount  of  cognitive  load  required),  the  amount  of  shared  knowledge  between  team  members 
(e.g.,  unique  roles),  time  pressure,  and  knowledge  uncertainty,  are  some  other  possibilities  that 
are  included  in  Warner,  Letsky,  and  Cowen’s  (2005)  model  that  may  affect  how  teams  transition 
through  the  collaboration  stages  and  macrocognitive  processes. 

Five  new  major  macrocognitive  processes  are  also  included  in  the  new  taxonomy:  1) 
Individual  Knowledge  Building,  2)  Team  Knowledge  Building,  3)  Developing  Shared  Problem 
Conceptualization,  4)  Team  Consensus  Development,  and  5)  Outcome  Appraisal  (see  Letsky,  et 
al.,  2007).  As  is  true  with  the  team  collaboration  stages,  these  processes  occur  dynamically, 
rather  than  sequentially.  While  both  the  stages  and  major  processes  described  in  the  model 
contribute  to  the  collaboration  process,  the  stages  are  not  cognitive  in  nature.  The  major 
processes,  however,  are  cognitive  processes  and  encompass  both  internalized  and  externalized 
processes  (see  Letsky,  et  al.,  2007). 

“Internalized  processes  are  those  higher-order  cognitive  processes  that  occur  at  the  individual  or  the  team 
level,  and  which  are  not  expressed  through  external  means  such  as  writing,  speaking,  gesture,  and  can  only  be 
measured  indirectly  via  qualitative  metrics  (e.g.,  cognitive  mapping,  think  aloud  protocols,  multi-dimensional 
scaling,  etc.),  or  surrogate  quantitative  metrics  (e.g.,  pupil  size,  galvanic  skin  response,  fMRI,  etc.). 
Externalized  processes  are  those  higher-order  cognitive  processes  that  occur  at  the  individual  or  the  team 
level,  and  which  are  associated  only  with  actions  that  are  observable  and  measurable  in  a  consistent,  reliable, 
repeatable  manner  or  through  the  conventions  of  the  subject  domain  having  standardized  meanings”  (Letsky, 
et  al.,  2007,  p.  7). 

In  addition  to  the  5  major  macrocognitive  processes,  Letsky  and  colleagues  (2007)  defined  and 
provided  descriptions  of  the  22  macrocognitive  subprocesses  that  nest  within  the  major  processes 
and  further  act  to  support  the  team  collaboration  stages.  Figure  2  is  taken  from  Letsky  and 
colleagues’  (2007)  paper  and  represents  the  taxonomy  they  proposed. 

GOALS  OF  THE  CURRENT  RESEARCH 


As  Letsky  and  colleagues  (2007)  stated,  one  of  the  lessons  learned  from  Warner,  Letsky,  and  Cowen’s  (2005) 
experiments  is  the  “the  need  to  understand  the  contribution  and  consistency  of  each  macrocognitive  process  across 
different  problem  domains. .  .which  will  provide  a  deeper  understanding  of  how  these  processes  impact  team 
collaboration  performance”  (p.  3).  In  re-  analyzing  Warner,  Letsky,  and  Cowen’s  (2005)  NEO  scenario  data,  the 
goal  was  to  provide  a  better  understanding  of  the  contribution  of  the  macrocognitive  major 
processes,  and  of  the  contribution  and  consistency  of  the  subprocesses  using  the  revised 
taxonomy,  which  should  facilitate  a  better  understanding  of  team  collaboration. 
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The  rationale  for  choosing  the  NEO  scenario  experimental  data  (Warner,  Letsky,  and 
Cowen,  2005)  was  four-fold.  The  first  reason  is  that  the  NEO  scenario,  as  opposed  to  the  murder 
mystery  scenario,  produced  communication  data  that  spanned  across  all  four  team  collaboration 
stages.  The  second  reason  is  that  since  the  communication  data  were  coded  using  an  earlier 
taxonomy  and  definitions  (see  Warner,  Letsky,  and  Cowen,  2005),  a  comparison  could  easily  be 
done  between  the  new  and  old  taxonomies.  The  third  reason  for  choosing  the  NEO  experimental 
data  involves  the  method  used  to  create  the  NEO  scenario  itself  (described  in  the  next  section). 
Finally,  as  more  and  more  teams  need  to  communicate  and  collaborate  on  problems  from 
different  locations  around  the  world,  results  from  studies  that  examine  asynchronous  and/or 
distributed  collaboration  will  need  to  play  a  more  important  role  in  the  development  of  models  of 
team  collaboration.  Thus,  since  the  original  NEO  experiment  (Warner,  Letsky,  and  Cowen, 
2005)  examined  face-to-face  versus  asynchronous-distributed  modes  of  communication,  it 
seemed  like  an  advantageous  set  of  data  to  re-examine. 

THE  CREATION  OF  TEAM  COLLABORATION  PROBLEM  SOLVING  SCENARIOS 


The  NEO:  Red  Cross  Rescue  Scenario  (Warner,  Wroblewski,  and  Shuck,  2004;  see 
appendices  A,  B,  C,  and  D)  was  developed  for  the  Office  of  Naval  Research  Collaboration  and 
Knowledge  Interoperability  program  in  order  to  facilitate  research  on  team  collaboration  and 
problem  solving.  The  researchers  developed  the  scenario  with  the  help  of  input  from  military 
personnel.  However,  both  military  and  nonmilitary  persons  can  solve  the  unclassified  scenario. 

Warner,  Wroblewski,  and  Shuck  (2004)  also  developed  a  scoring  matrix  for  the  final  NEO 
plan  (see  appendix  E).  They  did  this  with  input  from  military  operational  personnel  who  had 
experience  in  actual  NEO  scenarios.  The  researchers  created  the  matrix  so  that  they  could  score 
each  team’s  final  plan,  and  this  score  would  then  serve  as  an  objective  measure  of  team 
performance  for  data  analysis. 

The  scenario  requires  participants  to  develop  a  course  of  action  (COA)  to  rescue  three  Red 
Cross  workers  trapped  in  a  church  basement  on  a  fictitious  South  Pacific  Island.  Required 
elements  of  the  final  plan  include  a  list  of  U.S.  forces,  transportation,  weapons,  event  timeline, 
and  a  detailed  explanation  of  the  plan.  Each  team  member  receives  a  Mission  Statement,  data 
about  available  U.S.  military  assets,  topographical  maps,  infonnation  about  hostile  forces  in  the 
area,  and  other  descriptors  of  the  island  (see  appendix  A).  An  experimenter  (at  the  beginning  of 
the  session)  assigns  each  team  member  an  “expertise”  in  one  of  the  three  following  areas:  1) 
local  intelligence,  2)  available  weapons,  or  3)  local  environmental  issues.  The  experimenter  also 
provides  each  team  member  with  the  information  relevant  to  his/her  “expertise”  (see  appendices 
B,  C,  and  D).  At  the  end  of  the  problem  solving  session,  participants  submit  the  final  plan.  While 
the  NEO  is  only  one  example  of  a  complex,  event-based  scenario  that  can  be  used  as  a 
foundation  for  team  collaboration  experiments  (see  also  Warner,  Burkman,  and  Biron,  in  press), 
it  seemed  ideal  to  use  as  a  basis  for  our  reanalysis  since  it  was  designed  specifically  to  facilitate 
team  collaboration. 
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METHOD 


PARTICIPANTS 


A  total  of  96  undergraduate  students  from  a  community  college  in  Southern  Maryland 
served  as  the  participants  for  the  original  NEO  experiment  (see  Warner,  Letsky,  and  Cowen, 
2005).  Participants  consisted  of  36  men  and  60  women,  between  the  ages  of  17  to  53  years  of 
age  (Mean  =  24.13  years,  SD  =  10.28).  Participants  in  this  study  received  $15  an  hour  to 
complete  the  task  (which  could  last  up  to  4  hr)  and  their  professor  gave  them  extra  credit  for 
their  participation. 

DESIGN 


The  experimental  design  used  in  the  original  NEO  experiment  (see  Warner,  Letsky,  and 
Cowen,  2005)  was  a  two  by  two  randomized  factorial  with  two  levels  of  each  of  the  two 
independent  variables:  1)  collaboration  mode  (face-to-face  versus  asynchronous-distributed)  and 
2)  knowledge  uncertainty  (static  versus  dynamic).  The  experimenter  randomly  assigned  students 
to  the  conditions.  Thirty-two  three-member  teams  completed  the  experiment  and  the  data 
collected  from  all  of  these  teams  was  the  basis  for  the  present  re-analysis  using  the  revised 
taxonomy  and  definitions  proposed  by  Letsky  and  colleagues  (2007). 

Independent  Variables.  The  independent  variables  were  the  same  as  those  of  the  original 
NEO  experimental  design  (see  Warner,  Letsky,  and  Cowen,  2005).  The  first  independent 
variable,  collaboration  mode,  consisted  of  two  levels:  face-to-face  and  asynchronous-distributed. 
The  face-to-face  condition  consisted  of  teams  of  three  participants  interacting  synchronously 
with  each  other  around  a  conference  table  using  verbal  communication.  The  asynchronous- 
distributed  condition  also  consisted  of  three  member  teams.  However,  the  team  members 
interacted  with  each  other  from  different  locations  (cubicles  separated  by  partitions  located  in  the 
same  lab)  and  there  was  a  minimum  time  delay  of  1  sec  between  text  responses.  All  team 
communication  was  through  a  text-based  forum  (i.e.,  modified  Electronic  Card  Wall  -  EWall; 
see  Keel,  2004,  2005)  hosted  on  IBM  laptops  through  an  intranet  server.  Figure  3  illustrates  the 
modified  EWall  and  the  four  rooms  that  team  members  used  to  solve  the  NEO  collaborative 
problem. 
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CASC  Phase  II  Experiment  CASC  Phase  II  Experiment 

Modified  Ewall  Display:  Collaboration  and  Knowledge  Building  Modified  Ewall  Display:  Consensus  Room 


CASC  Phase  II  Experiment 
Modified  Ewall  Display:  Holding  Room 


CASC  Phase  II  Experiment 
Modified  Ewall  Display:  Final  Planning 


Figure  3:  Screen  Captures  from  Modified  EWall  Program 

Collaboration  and  knowledge  building  was  done  in  the  first  display  and  all  team  members 
could  see  all  current  text  responses  (i.e.,  up  to  the  most  recent  25  text  cards)  from  other  team 
members.  Cards  created  prior  to  the  25  most  recent  would  be  placed  in  the  Holding  Room  where 
they  could  be  viewed  at  any  time.  Team  consensus  was  worked  on  in  the  Consensus  Room 
where  solution  options  from  each  team  member  could  be  viewed  while  solutions  were  being 
developed.  The  team  developed  the  final  plan  in  the  Final  Planning  Room  and  submitted  the  plan 
at  the  end  of  the  session. 

Knowledge  uncertainly  served  as  the  second  independent  variable.  The  first  level  of  the 
variable  was  static  knowledge,  where  all  infonnation  (e.g.,  background  information,  rebels, 
military  assets,  and  weapons  or  environment  expertise  information)  remains  the  same  throughout 
the  collaborative  NEO  scenario  problem.  During  the  dynamic  knowledge  condition,  30  min  into 
the  collaboration  portion  of  the  session,  the  experimenter  provided  an  “update”  in  which  selected 
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information  (e.g.,  rebel  location  and  weather)  was  altered  (see  Warner,  Letsky,  and  Cowen, 
2005). 

Dependent  Variables.  Warner,  Letsky,  and  Cowen’s  (2005)  original  NEO  experiment 
included  five  dependent  variables:  (1)  score  on  the  final  plan,  (2)  time  to  complete  the  task,  (3) 
time  spent  in  each  of  the  collaboration  stages  and  macrocognitive  processes,  (4)  frequencies  of 
utterances  in  each  of  the  stages  and  processes,  and  (5)  transition  probabilities  between  the  stages 
and  processes  (see  Warner,  Letsky,  and  Cowen,  2005).  The  current  re-analysis  using  the  new 
taxonomy  and  definitions  (see  Letsky,  et  ah,  2007)  focused  on  the  time  spent  in  each  of  the  team 
collaboration  stages  and  macrocognitive  processes  and  the  transition  probabilities  between  the 
stages  and  major  processes. 

PROCEDURES 


Original  NEO  Experimental  Procedure.  The  original  procedure  used  in  the  NEO 
experiment  by  Warner  and  colleagues  (2005)  was  as  follows.  The  experimenter  instructed  each 
of  the  32  teams,  each  consisting  of  3  participants,  to  work  as  quickly  and  accurately  as  possible 
to  develop  a  COA  for  a  collaborative  problem-solving  task,  the  NEO  scenario  (see  appendices  A, 
B,  C,  and  D;  see  also  Warner,  Letsky,  and  Shuck,  2004).  Participants  all  completed  consent 
forms  before  beginning  the  experiment. 

Each  participant  was  first  given  instructions  (see  appendix  F)  which  included  a  brief 
description  of  the  problem  they  were  about  to  solve,  the  amount  of  time  they  would  have  to 
complete  the  task,  and  were  given  the  opportunity  to  ask  questions  about  the  task.  Participants 
were  then  given  20  min  to  individually  read  the  mission  statement,  background  information,  and 
randomly  assigned  expertise  infonnation  (either  weapons,  intelligence,  or  environmental).  The 
experimenter  then  assigned  one  of  the  team  members  to  act  as  the  scribe  during  the  experiment. 
The  scribe’s  task  was  to  fill  out  the  final  plan.  The  teams  then  had  a  maximum  of  60  min  to 
collaborate  in  either  a  face-to-face  or  an  asynchronous-distributed  manner  and  to  fill  out  the  final 
COA  plan.  The  final  plan  included  the  following  sections:  personnel,  transportation,  weapons, 
critical  times,  and  the  detailed  plan  (for  example,  see  appendix  G). 

Team  members  assigned  to  the  dynamic  condition  had  the  updated  intelligence  and  weather 
information  presented  to  them  30  min  into  the  session.  For  the  face-to-face  condition,  the 
experimenter  presented  the  team  a  typed  update  that  included  the  new  intelligence  and  weather 
information  (see  appendix  H).  The  asynchronous-distributed  condition  received  the  same  update 
(also  presented  30  min  into  the  session)  via  the  PC  server  on  their  IBM  laptop  workstations. 

All  of  the  communication  that  occurred  in  the  face-to-face  condition  was  audio  and  video 
recorded  and  timed  tagged  for  later  transcription;  whereas,  all  the  asynchronous-distributed  text 
was  recorded,  time  tagged,  and  saved  via  the  server.  After  all  teams  had  been  run  through  the 
experiment,  the  communication  data  were  coded  by  two  independent  raters  on  the  basis  of  both 
the  team  collaboration  stages  and  the  macrocognitive  processes  based  on  Warner,  Letsky,  and 
Cowen’s  (2005)  then  current  definitions.  (This  process  will  be  described  in-depth  under  the 
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section  of  this  report  describing  the  procedure  used  for  the  reanalysis.)  All  of  the  data  collected 
by  Warner,  Letsky,  and  Cowen  (2005)  was  analyzed  and  the  results  are  summarized  in  their 
paper. 

Procedure  for  the  Scoring  of  the  Final  Plan  for  the  NEO  Scenario.  As  each  team  worked  to 
solve  the  solution  for  the  NEO  scenario,  they  were  instructed  to  write  out  a  final  plan  listing 
assets  used  during  the  operation  (personnel,  transportation,  and  weapons),  critical  times 
calculated  (times  of  onset,  contact,  evacuation,  and  return  to  base  or  ship),  and  a  detailed  plan 
describing  exactly  how  the  mission  would  be  carried  out.  The  experimenter  also  gave  the  teams 
an  opportunity  to  add  any  additional  comments  regarding  their  final  plan.  The  form  in  appendix 
G  is  an  example  of  an  actual  team’s  final  plan  completed  in  the  previous  face-to-face  study. 

Using  the  example  in  appendix  G,  an  explanation  of  the  scoring  process  with  the  aid  of 
examples  may  offer  a  greater  understanding  of  how  the  researchers  scored  each  of  the  final 
plans.  In  the  first  section,  the  team  listed  the  personnel  they  would  be  using  on  this  mission.  The 
team  received  full  points  for  this  portion  because  they  included  all  the  necessary  information. 
For  the  transportation  section,  the  team  noted  that  they  would  be  using  the  Seahawk  and  a  C-130 
to  refuel  the  Seahawk.  They  lost  two  points  for  this  portion  because  they  chose  to  refuel  their 
aircraft;  however,  there  were  other  aircraft  options  that  did  not  require  the  added  risk  and  time 
needed  to  refuel.  In  the  third  box,  the  team  listed  the  weapons  they  planned  to  use  during  the 
mission.  The  team  in  the  example  properly  listed  all  the  necessary  weapons  and,  therefore, 
received  full  points  for  this  section. 

The  final  three  sections  of  the  final  plan  move  beyond  listing  the  assets  and  materials  and 
into  devising  a  well-constructed  plan  of  action.  The  fourth  section  required  the  team  to  list  the 
four  critical  times  of  the  mission.  Onset  of  operation  was  defined  as  the  time  at  which  the 
military  leaves  the  base  or  ship.  Contact  referred  to  the  time  the  military  made  contact  with  the 
Red  Cross  Workers  trapped  in  the  church.  The  team  also  needed  to  include  when  the  military 
would  evacuate  the  church  and  the  time  at  which  they  would  return  to  the  base  or  ship.  In  the 
example  in  appendix  G,  the  time  of  contact  was  listed  as  5:06  a.m.  and  the  time  of  evacuation 
was  scheduled  to  be  5:45.  According  to  the  team’s  plan,  the  military  would  be  in  the  church  for 
more  than  half  an  hour.  However,  time  was  of  the  essence,  and  once  the  military  entered  the 
church,  they  should  have  moved  as  quickly  as  possible  and  left  the  area  within  30  min. 
Therefore,  the  team  lost  another  two  points  for  this  violation. 

In  the  detailed  plan,  the  team  provided  a  description  of  the  mission.  They  also  provided 
additional  comments  that  proved  beneficial  to  their  final  score.  In  the  detailed  plan,  the  team 
described  how  the  Red  Cross  team  would  be  hoisted  from  the  ground  and  flown  back  to  the  ship. 
The  problem  with  this  part  of  the  team’s  solution  was  that  the  team  failed  to  avoid  detection 
because  flying  over  the  island  would  certainly  allow  the  rebels  to  hear  and  see  the  military 
aircraft,  giving  them  an  opportunity  to  attack.  In  the  additional  comments  section,  the  team 
mentioned  that  the  flight  was  performed  at  night  to  “avoid  detection”.  Avoiding  detection  is 
imperative  to  any  Special  Operations  mission,  and  the  Special  Forces  of  each  branch  are 
specifically  trained  in  this  area.  Since  the  team  addressed  avoiding  detection  but  failed  to  do  so 
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by  flying  so  close  to  the  island,  they  only  lost  two  points.  If  they  had  not  made  this  additional 
note,  one  additional  point  would  have  been  deducted,  as  they  would  have  failed  to  address 
possible  detection.  In  addition,  the  team  addressed  giving  medical  attention  to  the  Red  Cross 
workers  once  they  were  in  the  aircraft  in  the  “additional  comments”  portion  of  the  plan.  If  they 
had  no  mentioned  this,  the  team  would  have  lost  an  additional  five  points,  since  aiding  the  Red 
Cross  workers  is  the  main  priority  in  this  situation. 

The  team’s  final  score  was  94  out  of  a  possible  100  points.  See  appendix  E  for  a  complete 
matrix  delineating  point  deductions  for  each  planning  card.  The  “ideal”  solution  is  shown  in 
appendix  I. 

Procedure  for  the  Re-analyses.  As  was  the  case  with  the  original  NEO  experimental  data, 
the  communication  data  needed  to  be  assessed  in  terms  of  the  team  collaboration  stages.  Once 
again,  two  independent  raters  coded  the  transcripts  for  the  team  collaboration  stages.  The  process 
began  with  each  of  the  new  raters  familiarizing  themselves  with  the  definitions  and  examples  for 
the  four  team  collaboration  stages.  Then  each  rater  went  through  a  subset  of  the  communication 
transcripts  (10  teams  total  -  5  face-to-face  and  5  asynchronous-distributed).  For  each  line  of 
communication  data  (utterance),  each  rater  independently  had  to  decide  which  of  the  four  team 
collaboration  stages  best  described  that  line  of  data  and  label/code  the  line  based  on  that 
decision.  The  interrater  reliability  between  the  two  raters  was  then  calculated  and  compared  with 
the  score  obtained  from  the  two  original  raters  and  found  to  be  statistically  equivalent  (reported 
in  the  Results  section).  Thus,  since  the  old  and  new  taxonomies  did  not  differ  in  terms  of  the 
team  collaboration  stages,  the  original  raters’  coded  data  were  used  for  the  remainder  of  the 
analyses. 

The  next  task  for  the  two  raters  was  to  go  through  the  new  taxonomy  and  definitions  (see 
Letsky,  et  ah,  2007)  and  make  sure  that  adequate  examples  existed  for  the  processes  that  carried 
over  from  the  earlier  taxonomy  or  were  added  for  all  of  the  new  definitions  for  both  the 
macrocognitive  major  processes  and  subprocesses  (for  all  definitions  and  examples,  see 
appendices  J  and  K).  The  examples  were  all  agreed  upon  by  both  raters  to  be  useful  for  future 
coding  based  on  the  definitions  and  were  taken  from  the  communication  transcripts  from  the 
original  NEO  experiment  (Warner,  Letsky,  and  Cowen,  2005).  For  three  of  the  macrocognitive 
subprocesses,  an  additional  step  was  taken  before  the  two  raters  began  coding  the  transcripts.  For 
these  three  subprocesses  (Storyboarding,  Mental  Simulation,  and  Building  Common  Ground), 
the  researchers  added  operational  definitions  that  could  be  used  to  measure  these  processes  given 
the  communication  data  (see  appendix  K).  These  measures,  given  the  lack  of  orthogonality  with 
the  other  subprocesses  and  their  method  of  calculation,  became  secondary  measures  since  they 
were  dependent  upon  other  subprocesses  for  their  measurement. 

The  transcripts  were  coded  independently  by  the  two  raters  once  they  had  gone  over  all  of 
the  definitions  and  agreed  upon  the  new  examples.  Each  rater  went  through  each  line  of  text  (that 
either  corresponded  to  transcribed  spoken  statements  or  typed  text  used  to  communicate  during 
the  experiment)  and,  on  the  first  pass  through,  determined  which  of  the  macrocognitive  major 
processes  to  assign.  In  order  to  accomplish  this,  raters  looked  at,  not  only  each  line  of  text 
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individually,  but  they  also  considered  the  context  (e.g.,  surrounding  lines  of  text  that  related  to 
the  same  topic)  as  well.  Based  on  the  definitions  and  examples,  the  raters  were  looking  for 
reasons  to  select  one  process  over  the  others.  Some  of  the  criteria  that  aided  the  raters  in  this 
process  included:  whether  or  not  the  line(s)  included  question  asking  and  answering,  whether  or 
not  someone  had  suggested  a  solution  option,  whether  or  not  the  team  was  discussing  goals  of 
the  problem,  and  whether  or  not  the  participant(s)  referred  to  a  visual  aid,  such  as  a  map  or 
drawing.  For  example,  if  all  three  team  members  were  involved  in  raising  and  discussing  a 
solution  to  the  problem,  this  falls  under  the  macrocognitive  major  process  of  Team  Consensus 
Development.  If  two  team  members  were  discussing  the  rules  they  must  follow  to  solve  the 
problem  or  the  goals  of  the  problem  (without  asking  questions),  raters  would  code  this  as 
Developing  Shared  Problem  Conceptualization.  Also  taken  into  consideration  was  how  many  of 
the  three  team  members  were  involved  in  discussing  the  current  topic  considering  that  several  of 
the  definitions  (e.g.,  Team  Knowledge  Building)  must  include  all  three  team  members  (see 
appendix  J),  whereas  others  do  not  (e.g.,  Individual  Knowledge  Building).  However,  if  after 
considering  all  of  the  five  macrocognitive  major  processes  a  rater  detennined  that  none  of  them 
accurately  reflected  a  line  of  text,  they  labeled  the  line  as  “miscellaneous”.  After  the  two  raters 
independently  coded  all  of  the  transcripts  for  the  macrocognitive  major  processes,  they  repeated 
the  same  process  for  the  subprocesses.  Once  this  was  completed,  they  began  the  process  of 
reconciling  the  lines  in  the  transcript  for  which  the  two  raters  disagreed  on  the  appropriate  code. 

In  order  to  reconcile  any  coding  differences  between  the  raters  in  the  transcripts,  the  two 
raters  convened  and  went  through  and  discussed  each  line  of  text  where  a  disagreement 
occurred.  For  each  of  these  instances,  raters  discussed  their  rationales  for  the  code  that  each  had 
chosen,  the  definition(s),  examples,  similar  occurrences  from  the  transcripts,  and  anything  else 
pertinent  until  a  consensus  was  reached  regarding  the  final  decision  as  to  what  the  line  should  be 
coded. 


RESULTS 

Warner,  Letsky,  and  Cowen  (2005),  in  their  original  analyses  used  a  2x2  ANOVA  to 
analyze  the  total  time  to  complete  the  task  across  collaboration  modes  and  the  level  of 
knowledge  uncertainty  along  with  the  accuracy  of  the  final  plan  and  the  frequency  of  utterances. 
The  researchers  found  that  there  were  no  significant  differences  between  the  face-to-face  and 
asynchronous-distributed  conditions  in  the  total  amount  of  time  to  complete  the  task,  F=0.26, 
p=0.62;  or  the  level  of  knowledge  uncertainty,  F=1.88,  p=0.18.  The  interaction  was  also  found 
not  to  be  significant,  F=0.13,  p=0.72.  There  was  no  significant  difference  in  the  accuracy  of  the 
final  plan  between  the  two  collaboration  modes,  F=2.37,  p=0.14,  or  knowledge  uncertainty, 
F=0.07,  p=0.80.  The  interaction  between  them  was  not  significant,  F=0.37,  p=0.55.  There  was  a 
significant  difference  found  between  the  face-to-face  and  the  asynchronous-distributed  teams, 
F=96.50,  p  <  0.001.  The  researchers  also  calculated  the  mean  numbers  of  utterances  across  the 
conditions.  For  the  face-to-face  teams,  the  mean  number  of  utterances  was  847.31,  whereas  it 
was  only  91.13  for  the  asynchronous-distributed  teams.  Once  again,  there  were  no  significant 
differences  found  regarding  the  variable  of  knowledge  uncertainty,  F=2.06,  p=0.16,  or  the 
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interaction,  F=2.73,  p=0.11.  These  results  were  all  reported  in  Warner,  Letsky,  and  Cowen’s 
(2005)  paper. 

THE  TEAM  COLLABORATION  STAGES 

As  was  mentioned  earlier,  the  interrater  reliability  for  the  team  collaboration  stages  between 
the  two  new  raters  (0.80)  was  not  found  to  be  significantly  different  from  the  reliability 
calculated  for  the  two  raters  used  for  the  original  analyses  (0.80).  Therefore,  the  original  data 
labels  were  used  and  the  percentage  of  total  time  spent  in  each  of  the  stages  was  calculated. 

A  2x2  ANOVA  was  used  to  re-analyze  the  percentage  of  total  time  spent  in  each  of  the  team 
collaboration  stages  across  collaboration  modes  and  the  level  of  knowledge  uncertainty.  There 
was  no  significant  main  effect  found  for  the  two  collaboration  modes,  F=1.54,  p=0.22.  There  was 
also  no  significant  difference  found  for  knowledge  uncertainty,  F=0.45,  p=0.77,  and  the 
interaction  was  not  significant,  F=0.76,  p=0.56.  As  for  the  univariate  results  for  each  of  the 
stages,  there  was  a  significant  difference  between  the  two  collaboration  modes  in  the  mean 
percentage  of  time  spent  in  each  collaboration  stage.  The  face-to-face  teams  spent  more  time  in 
Knowledge  Construction  than  the  asynchronous-distributed  teams,  F=6.51,  p  <  0.01.  The 
asynchronous-distributed  teams  spent  significantly  more  time  in  the  Collaborative  Team 
Problem  Solving  stage  than  the  face-to-face  teams,  F=5.60,  p  <  0.05.  There  were  no  other 
significant  main  or  interaction  effects  for  collaboration  stages.  These  dependent  variable 
univariate  results  were  also  reported  in  Warner,  Letsky,  and  Cowen’s  (2005)  paper. 

THE  MACROCOGNITIVE  MAJOR  PROCESSES 

The  interrater  reliability  between  the  two  raters  based  on  their  coding  of  the  macrocognitive 
major  processes  was  calculated.  The  interrater  reliability  was  0.82.  Then,  the  percentage  of  time 
spent  in  each  of  the  major  macrocognitive  processes  was  calculated  and  used  for  the  remainder 
of  the  analyses,  as  was  done  with  the  team  collaboration  stages. 

A  2x2  ANOVA  was  used  to  analyze  the  percentage  of  total  time  spent  in  each  of  the 
macrocognitive  major  processes  across  collaboration  modes  and  the  level  of  knowledge 
uncertainty  (see  figure  4).  Overall,  the  only  significant  difference  was  found  between  the  two 
collaboration  modes,  F=15.88,  p  <  0.001.  In  comparisons  between  the  major  macrocognitive 
processes,  the  only  significant  differences  were  found  across  the  collaboration  modes.  Face-to- 
face  teams  spent  significantly  more  time  than  asynchronous-distributed  teams  in  the  Team 
Knowledge  Building  process,  F=29.53,  p  <  0.001.  The  asynchronous-distributed  teams  were 
significantly  more  likely  to  spend  a  greater  percentage  of  time  engaged  in  the  Developing  Shared 
Problem  Conceptualization  process  than  face-to-face  teams,  F=8.62,  p  <  0.01.  No  other 
significant  differences  were  found. 

The  data  were  then  transformed  to  examine  how  the  major  processes  were  nested  within 
each  of  the  team  collaboration  stages.  A  2x2  ANOVA  was  used  to  analyze  the  percentage  of 
total  time  spent  in  each  of  the  major  processes  during  each  of  the  collaboration  stages  across 
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collaboration  modes  and  the  level  of  knowledge  uncertainty.  There  was  a  significant  difference 
across  collaboration  mode,  F=6.48,  p  <  0.01.  There  was  no  significant  difference  for  knowledge 
uncertainty,  F=0.96,  p=0.55.  The  interaction  was  not  significant,  F=0.61,  p=0.83. 
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Figure  4:  Percentage  of  Time  Spent  in  Major  Macrocognitive  Processes  by  Collaboration  Mode 

During  Knowledge  Construction,  Individual  Knowledge  Building,  F=54.23,  p  <  0.001, 
Team  Knowledge  Building,  F=58.47,  p  <  0.001;  Developing  Shared  Problem  Conceptualization, 
F=21.75,  p  <  0.001;  and  Team  Consensus  Development,  F=68.56,  p  <  0.001,  were  found  to 
occur  a  significant  percentage  of  the  time.  During  Collaborative  Team  Problem  Solving,  all  five 
macrocognitive  processes  were  significant:  Individual  Knowledge  Building,  F=32.51,  p  <  0.001; 
Team  Knowledge  Building,  F=20.74,  p  <  0.001;  Developing  Shared  Problem  Conceptualization, 
F=17.52,  p  <  0.001;  Team  Consensus  Development,  F=785.17,  p  <  0.001;  and  Outcome 
Appraisal,  F=4.10,  p  <  0.05.  During  Team  Consensus,  Team  Consensus  Development,  F=18.45, 
p  <  0.001,  and  Outcome  Appraisal,  F=4.04,  p  <  0.05,  were  both  significant.  Finally,  during 
Outcome,  Evaluation,  and  Revision  only  Team  Consensus  Development,  F=9.69,  p  <  0.01,  was 
significant.  These  results  are  summarized  in  figure  5. 
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Figure  5:  Taxonomy  of  Macrocognitive  Major  Processes  and  the  Team  Collaboration  Stages 

The  univariate  results  for  this  analysis  also  produced  some  significant  differences.  During 
the  Knowledge  Construction  stage,  the  face-to-face  teams  spent  significantly  more  time  in  the 
Team  Knowledge  Building  process  than  the  asynchronous-distributed  teams,  F=39.59,  p  <  0.001. 
During  the  Collaborative  Team  Problem  Solving  stage,  the  face-to-face  teams  again  spent 
significantly  more  time  in  the  Team  Knowledge  Building  process  than  the  asynchronous- 
distributed  teams,  F=7.47,  p  <  0.01.  However,  during  both  the  Knowledge  Construction,  F=8.24, 
p  <  0.01,  and  Collaborative  Team  Problem  Solving  stages,  F=6.44,  p  <  0.05,  asynchronous- 
distributed  teams  spent  significantly  more  time  in  the  Developing  Shared  Problem 
Conceptualization  process  than  the  face-to-face  teams.  There  were  no  other  significant 
differences. 

THE  MACROCOGNITIVE  SUBPROCESSES 

The  interrater  reliability  between  the  two  raters  based  on  their  coding  of  the  macrocognitive 
subprocesses  was  calculated.  The  interrater  reliability  was  0.71.  A  2x2  ANOVA  was  used  to 
analyze  the  percentage  of  total  time  spent  in  each  of  the  macrocognitive  subprocesses 
collaboration  modes  and  the  level  of  knowledge  uncertainty.  There  was  a  significant  difference 
between  the  two  collaboration  modes,  F=7.03,  p  <  0.01.  There  was  no  significant  difference  for 
knowledge  uncertainty,  F=1 . 10,  p=0.45.  The  interaction  was  not  significant,  F=1.29,  p=0.35.  The 
following  subprocesses  were  significant:  Iterative  Infonnation  Collection,  F=44.47,  p  <  0.001; 
Individual  Task  Knowledge  Development,  F=186.87,  p  <  0.001;  Individual  Mental  Model 
Development,  F=28.67,  p  <  0.001;  Sharing  Unique  Knowledge,  F=6.68,  p  <  0.05;  Knowledge 
Interoperability,  F=141.96,  p  <  0.001;  Visualization  and  Representation  of  Meaning,  F=32.95,  p 
<  0.001;  Knowledge  Sharing,  F=66.46,  p  <  0.001;  Knowledge  Transfer,  F=31.35,  p  <  0.001; 
Team  Shared  Understanding,  F=13.63,  p  <  0.001;  Critical  Thinking,  F=8.42,  p  <  0.01;  Solution 
Option  Generation,  F=36.87,  p  <  0.001;  and  Feedback  Interpretation,  F=7.67,  p  <  0.01  (see 
figure  6). 

There  were  a  number  of  significant  ANOVA  univariate  results  for  each  of  the  subprocesses 
(see  figure  6).  For  Individual  Task  Knowledge  Development,  there  was  a  significant  difference 
depending  on  the  collaboration  mode,  F=17.32,  p  <  0.001.  Teams  in  the  asynchronous- 
distributed  condition  spent  more  time  in  the  Individual  Task  Knowledge  Development 
subprocess  that  teams  in  the  face-to-face  condition.  For  Knowledge  Interoperability,  face-to-face 
teams  spent  more  time  than  asynchronous-distributed  teams,  F=52.16,  p  <  0.001.  There  was  a 
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significant  difference  depending  on  the  collaboration  mode  for  Visualization  and  Representation 
of  Meaning,  F=18.62,  p  <  0.001.  Face-to-face  teams  spent  more  time  in  this  subprocess  than 
teams  in  the  asynchronous-distributed  condition.  For  the  subprocess  of  Solution  Option 
Generation  (see  figure  7),  asynchronous-distributed  teams  spent  more  time  than  face-to-face 
teams,  F=1 5.  11,  p  <  0  .001.  Teams  in  the  static  knowledge  condition  spent  more  time  in  the 
subprocess  of  Solution  Option  Generation  than  teams  in  the  dynamic  knowledge  condition, 
F=6.50,  p  <  0.05.  The  interaction  between  knowledge  uncertainty  condition  and  the  two 
collaboration  modes  was  also  significant,  F=5.89,  p  <  0.05. 


Figure  6:  Percentage  of  Time  Spent  in  Macrocognitive  Subprocesses  by  Collaboration  Mode 
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Figure  7:  Percentage  of  Time  Spent  in  Solution  Option  Generation  across 

Two  Independent  Variables 

As  for  the  three  secondary  measures  which  were  calculated  based  on  the  other 
macrocognitive  subprocesses  (see  appendix  K),  these  were  analyzed  based  on  the  mean 
percentage  of  the  total  number  of  utterances.  A  2x2  ANOVA  was  performed  for  these  three 
dependent  variables  across  knowledge  uncertainty  and  collaboration  mode.  Face-to-face  teams 
spend  significantly  more  time  engaged  in  the  subprocess  of  Building  Common  Ground  than 
asynchronous-distributed  teams,  F=21.58,  p  <  0.001.  Asynchronous-distributed  teams  spent 
significantly  more  time  in  the  Storyboarding  subprocess  than  face-to-face  teams,  F=19.93,  p  < 
0.001.  There  were  no  other  significant  differences.  The  significant  results  for  the  secondary 
measures  are  shown  in  figure  8. 
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Figure  8:  Three  Macrocognitive  Subprocesses  across  Collaboration  Modes 

The  data  were  then  transformed  to  examine  how  the  macrocognitive  subprocesses  were 
nested  within  each  of  the  major  processes.  For  this  analysis,  only  the  subprocesses  that  were 
already  found  to  be  significant  overall  were  included  (see  figure  6).  A  2x2  ANOVA  was  then 
used  to  analyze  the  percentage  of  total  time  spent  in  each  of  the  macrocognitive  subprocesses 
that  were  found  to  be  significant  within  each  of  the  five  major  processes  across  collaboration 
modes  and  the  level  of  knowledge  uncertainty.  There  was  a  significant  difference  found  for 
collaboration  mode,  F=359.98,  p  <  0.05.  There  was  no  significant  main  effect  for  knowledge 
uncertainty,  F=79.52,  p=0.09.  The  interaction  between  them  was  also  not  significant,  F=200.85, 
p=0.06.  During  the  Individual  Knowledge  Building  major  process,  the  following  subprocesses 
were  significant:  Iterative  Infonnation  Collection,  F=6.24,  p  <  0.05;  Individual  Task  Knowledge 
Development,  F=65.27,  p  <  0.001;  Individual  Mental  Model  Development,  F=4.12,  p  <  0.05; 
Sharing  Unique  Knowledge,  F=5.81,  p  <  0.05;  Knowledge  Interoperability,  F=23.24,  p  <  0.001; 
Visualization  and  Representation  of  Meaning,  F=9.88,  p  <  0.01;  Knowledge  Sharing,  F=40.82,  p 
<  0.001;  Knowledge  Transfer,  F=6.15,  p  <  0.05;  Team  Shared  Understanding,  F=4.68,  p  <  0.05; 
Building  Common  Ground,  F=28.86,  p  <  0.001;  and  Storyboarding,  F=17.05,  p  <  0.001.  Within 
the  macrocognitive  major  process  Team  Knowledge  Building,  the  following  subprocesses  were 
found  to  be  significant:  Iterative  Information  Collection,  F=5.12,  p  <  0.05;  Individual  Task 
Knowledge  Development,  F=52.79,  p  <  0.001;  Individual  Mental  Model  Development,  F=14.51, 
p  <  0.001;  Knowledge  Interoperability,  F=21.62,  p  <  0.001;  Visualization  and  Representation  of 
Meaning,  F=16.03,  p  <  0.001;  Knowledge  Sharing,  F=1 1 .79,  p  <  0.01;  Team  Shared 
Understanding,  F=  1 1.58,  p  <  0.01;  Building  Common  Ground,  F=25.73,  p  <  0.001;  and 
Storyboarding,  F=46.39,  p  <  0.001.  During  the  Developing  Shared  Problem  Conceptualization 
macrocognitive  major  process,  the  following  subprocesses  were  significant:  Iterative  Information 
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Collection,  F=4.84,  p  <  0.05;  Individual  Task  Knowledge  Development,  F=39.87,  p  <  0.001; 
Knowledge  Interoperability,  F=6.43,  p  <  0.05;  Knowledge  Sharing,  F=4.10,  p  <  0.05;  Building 
Common  Ground,  F=7.92,  p  <  0.01;  and  Storyboarding,  F=9.20,  p  <  0.01.  Within  the  Team 
Consensus  Development  macrocognitive  major  process,  the  following  subprocesses  were 
significant:  Iterative  Information  Collection,  F=5 1.19,  p  <  0.001;  Individual  Task  Knowledge 
Development,  F=50.72,  p  <  0.001;  Individual  Mental  Model  Development,  F=10.83,  p  <  0.01; 
Sharing  Unique  Knowledge,  F=5.47,  p  <  0.05;  Knowledge  Interoperability,  F=77.82,  p  <  0.001; 
Visualization  and  Representation  of  Meaning,  F=19.28,  p  <  0.001;  Knowledge  Sharing,  F=20.24, 
p  <  0.001;  Knowledge  Transfer,  F=21.04,  p  <  0.001;  Team  Shared  Understanding,  F=6.49,  p  < 
0.05;  Critical  Thinking,  F=4.78,  p  <  0.05;  Solution  Option  Generation,  F=29.93,  p  <  0.001; 
Feedback  Interpretation,  F=5.02,  p  <  0.05;  Building  Common  Ground,  F=127.92,  p  <  0.001;  and 
Storyboarding,  F=1 80. 1 8,  p  <  0.001.  Finally,  within  the  Outcome  Appraisal  macrocognitive 
major  process,  only  two  subprocesses  were  found  to  be  significant:  Individual  Task  Knowledge 
Development,  F=4.30,  p  <  0.05,  and  Building  Common  Ground,  F=4.24,  p  <  0.05.  These  results 
are  summarized  in  figure  9. 
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Figure  9:  Taxonomy  of  Macrocognitive  Major  Processes  and  Subprocesses 

Regarding  the  univariate  results  for  this  analysis,  there  were  a  number  of  significant 
findings.  Within  Individual  Knowledge  Building,  there  was  a  significant  difference  across 
collaboration  modes  for  Knowledge  Interoperability,  F=15.69,  p  <  0.001.  Face-to-face  teams 
spent  more  time  in  the  Knowledge  Interoperability  subprocess  during  the  macrocognitive  major 
process  of  Individual  Knowledge  Building  than  asynchronous-distributed  teams.  Face-to-face 
teams  spent  more  time  in  Visualization  and  Representation  of  Meaning  than  asynchronous- 
distributed  teams  while  engaging  in  Individual  Knowledge  Building,  F=9.46,  p  <  0.01.  Face-to- 
face  teams  also  spent  more  time  engaged  in  Building  Common  Ground  during  Individual 
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Knowledge  Building  than  asynchronous-distributed  teams,  F=1 1 .66,  p  <  0.01.  Asynchronous- 
distributed  teams,  however,  spent  more  time  engaged  in  Knowledge  Sharing  during  Individual 
Knowledge  Building  than  face-to-face  teams,  F=8.32,  p  <  0.01.  There  were  no  significant 
differences  across  knowledge  uncertainty  during  Individual  Knowledge  Building. 

For  the  macrocognitive  major  process  of  Team  Knowledge  Building  there  were  significant 
differences  for  the  univariate  results  as  well.  For  Team  Knowledge  Building,  there  was  a 
significant  difference  across  collaboration  mode  for  Individual  Task  Knowledge  Development, 
F=21.80,  p  <  0.001.  While  engaging  in  Team  Knowledge  Building,  face-to-face  teams  spent 
more  time  in  the  subprocess  of  Individual  Task  Knowledge  Development  than  asynchronous- 
distributed  teams.  Within  Team  Knowledge  Building,  face-to-face  teams  spent  more  time 
engaged  in  Individual  Mental  Model  Development  than  asynchronous-distributed  teams, 
F=  14.51,  p  <  0.001.  While  teams  were  engaged  in  Team  Knowledge  Building,  face-to-face 
teams  also  spent  more  time  than  asynchronous-distributed  teams  engaged  in  Knowledge 
Interoperability,  F=18.30,  p  <  0.001.  For  Team  Knowledge  Building,  there  was  a  significant 
difference  for  Visualization  and  Representation  of  Meaning  across  collaboration  mode,  F=16.03, 
p  <  0.001.  Within  Team  Knowledge  Building,  face-to-face  teams  spent  more  time  engaged  in 
Visualization  and  Representation  of  Meaning  than  asynchronous-distributed  teams.  Face-to-face 
teams  also  spent  more  time  in  Knowledge  Sharing  than  asynchronous-distributed  teams  during 
Team  Knowledge  Building,  F=1 1 .79,  p  <  0.01,  and  in  Building  Common  Ground,  F=9,63,  p  < 
0.01.  During  asynchronous-distributed  teams’  collaboration,  more  time  was  spent  during  Team 
Knowledge  Building  in  the  subprocess  of  Storyboarding,  F=37.95,  p  <  0.001.  Finally,  during 
Team  Knowledge  Building,  there  was  a  significant  difference  across  collaboration  mode  for 
Team  Shared  Understanding,  F=  11.57,  p  <  0.01.  Face-to-face  teams  spent  more  time  in  Team 
Shared  Understanding  than  asynchronous-distributed  teams.  During  Team  Knowledge  Building, 
no  significant  differences  were  found  across  knowledge  uncertainty. 

Within  Developing  Shared  Problem  Conceptualization,  there  were  only  two  significant 
differences  found  across  collaboration  mode.  During  Developing  Shared  Problem 
Conceptualization,  asynchronous-distributed  teams  spent  more  time  than  face-to-face  teams 
engaged  in  Individual  Task  Knowledge  Development,  F=18.93,  p  <  0.001,  and  in  Storyboarding, 
F=6.05,  p  <  0.05.  No  significant  differences  were  found  across  knowledge  uncertainty  for  the 
macrocognitive  major  process  of  Developing  Shared  Problem  Conceptualization. 

During  the  macrocognitive  major  process  of  Team  Consensus  Development,  there  were  also 
a  number  of  significant  differences.  Within  Team  Consensus  Development,  there  was  a 
significant  difference  across  collaboration  mode  for  Iterative  Infonnation  Collection,  F=4.38,  p  < 
0.05.  Asynchronous-distributed  teams  spent  more  time  in  Iterative  Information  Collection  than 
face-to-face  teams.  During  Team  Consensus  Development,  asynchronous-distributed  teams  spent 
more  time  than  face-to-face  teams  engaged  in  Individual  Task  Knowledge  Development, 
F=16.57,  p  <  0.001,  and  in  Storyboarding,  F=15.24,  p  <  0.001.  Within  the  Team  Consensus 
Development  major  process,  face-to-face  teams  spent  more  time  than  asynchronous-distributed 
teams  engaged  in  Knowledge  Interoperability,  F=13.08,  p  <  0.01,  and  in  Building  Common 
Ground,  F=20.29,  p  <  0.001.  Face-to-face  teams  also  spent  more  time  than  asynchronous- 
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distributed  teams  engaged  in  Visualization  and  Representation  of  Meaning  during  Team 
Consensus  Development,  F=6.30,  p  <  0.05.  Within  Team  Consensus  Development,  there  was  a 
significant  difference  for  Solution  Option  Generation  across  both  collaboration  mode,  F=15.17,  p 
<  0.001,  and  knowledge  uncertainty,  F=8.78,  p  <  0.01,  and  the  interaction  was  significant, 
F=6.43,  p  <  0.05.  During  Team  Consensus  Development,  asynchronous-distributed  teams  spent 
more  time  engaged  in  Solution  Option  Generation  than  face-to-face  teams,  and  static  knowledge 
teams  spent  more  time  in  Solution  Option  Generation  than  dynamic  teams.  Asynchronous- 
distributed  teams  spent  more  time  engaged  in  Storyboarding  than  face-to-face  teams  during 
Team  Consensus  Development,  F=15.24,  p  <  0.001. 

Finally,  there  were  no  significant  differences  found  across  collaboration  mode  or  knowledge 
uncertainty  for  any  of  the  subprocesses  within  the  macrocognitive  major  process  of  Outcome 
Appraisal.  There  were  also  no  significant  interactions  found  for  the  subprocesses  during 
Outcome  Appraisal. 

A  NEW  TAXONOMY 


The  results  summarized  in  figures  5  and  9  combined  still  do  not  address  the  issue  of  which 
of  the  macrocognitive  subprocesses  support  each  of  the  team  collaboration  stages  during 
problem  solving.  In  order  to  complete  a  new  taxonomy  based  on  the  revised  definitions  (Letsky, 
et  ah,  2007),  and  the  re-analyses  presented  in  this  report,  the  data  were  transformed  to  examine 
which  of  the  macrocognitive  subprocesses  were  engaged  in  a  significant  percent  of  time  in  each 
of  the  team  collaboration  stages.  A  2x2  ANOVA  was  then  used  to  analyze  the  percentage  of  total 
time  spent  in  each  of  the  macrocognitive  subprocesses  within  each  of  the  team  collaboration 
stages  across  collaboration  modes  and  the  level  of  knowledge  uncertainty.  For  this  analysis,  only 
the  subprocesses  that  were  already  found  to  be  significant  overall  were  included  (see  figure  6). 
Overall,  there  was  no  significant  difference  between  the  two  collaboration  modes,  F=9.10, 
p=0.26,  or  the  knowledge  uncertainty  conditions,  F=0.63,  p=0.78.  The  interaction  was  also  not 
significant,  F=1.48,  p=0.58.  The  following  subprocesses  were  significant  during  the  Knowledge 
Construction  team  collaboration  stage:  Iterative  Infonnation  Collection,  F=9.97,  p  <  0.01; 
Individual  Task  Knowledge  Development,  F=135.19,  p  <  0.001;  Individual  Mental  Model 
Development,  F=10.31,  p  <  0.01;  Sharing  Unique  Knowledge,  F=5.033,  p  <  0.05;  Knowledge 
Interoperability,  F=67.34,  p  <  0.001;  Visualization  and  Representation  of  Meaning,  F=20.80,  p  < 
0.001;  Knowledge  Sharing,  F=32.66,  p  <  0.001;  Knowledge  Transfer,  F=9.35,  p  <  0.01;  Team 
Shared  Understanding,  F=12.81,  p  <  0.01;  Building  Common  Ground,  F=102.16,  p  <  0.001;  and 
Storyboarding,  F=40.20,  p  <  0.001.  The  following  subprocesses  were  significant  during  the 
Collaborative  Team  Problem  Solving  stage:  Iterative  Information  Collection,  F=36.84,  p  < 
0.001;  Individual  Task  Knowledge  Development,  F=132.38,  p  <  0.001;  Individual  Mental  Model 
Development,  F=44.49,  p  <  0.001;  Sharing  Unique  Knowledge,  F=7.11,  p  <  0.05;  Knowledge 
Interoperability,  F=127.26,  p  <  0.001;  Visualization  and  Representation  of  Meaning,  F=23.97,  p 
<  0.001;  Knowledge  Sharing,  F=35.96,  p  <  0.001;  Knowledge  Transfer,  F=30.21,  p  <  0.001; 
Team  Shared  Understanding,  F=9,50,  p  <  0.01;  Critical  Thinking,  F=7.84,  p  <  0.01;  Solution 
Option  Generation,  F=31.68,  p  <  0.001;  Feedback  Interpretation,  F=7.14,  p  <  0.05;  Building 
Common  Ground,  F=133.09,  p  <  0.001;  and  Storyboarding,  F=148.76,  p  <  0.001.  The  following 
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subprocesses  were  significant  during  the  Team  Consensus  stage:  Individual  Task  Knowledge 
Development,  F=4.05,  p  <  0.05;  Knowledge  Interoperability,  F=26.26,  p  <  0.001;  Visualization 
and  Representation  of  Meaning,  F=4.40,  p  <  0.05;  Building  Common  Ground,  F=20.02,  p  <  01; 
and  Storyboarding,  F=6.00,  p  <  0.05.  The  following  subprocesses  were  significant  during  the 
Outcome,  Evaluation,  and  Revision  stage:  Knowledge  Interoperability,  F=4.08,  p  <  0.05; 
Visualization  and  Representation  of  Meaning,  F=4.05,  p  <  0.05;  and  Building  Common  Ground, 
F=4.91,  p  <  0.05. 


There  were  also  some  significant  findings  within  the  univariate  results  for  this  analysis. 
During  Knowledge  Construction,  there  was  a  significant  difference  across  collaboration  modes 
for  Knowledge  Interoperability,  F=50.55,  p  <  0.001.  Face-to-face  teams  spent  more  time  in  the 
Knowledge  Interoperability  subprocess  within  the  team  collaboration  stage  of  Knowledge 
Construction  than  asynchronous-distributed  teams.  Face-to-face  teams  also  spent  more  time  in 
Visualization  and  Representation  of  Meaning  than  asynchronous-distributed  teams  during  the 
Knowledge  Construction  stage,  F=20.77,  p  <  0.001.  Within  Knowledge  Construction,  face-to- 
face  teams  spent  more  time  than  asynchronous-distributed  teams  engaged  in  both  Knowledge 
Transfer,  F=8.25,  p  <  0.01,  and  Team  Shared  Understanding,  F=12.81,  p  <  0.01.  During  the 
Collaborative  Team  Problem  Solving  stage,  significant  differences  were  found  for  the  following 
subprocesses:  Individual  Task  Knowledge  Development,  F=20.26,  p  <  0.001;  Individual  Mental 
Model  Development,  F=6.48,  p  <  0.05;  Knowledge  Interoperability,  F=42.88,  p  <  0.001; 
Visualization  and  Representation  of  Meaning,  F=12.75,  p  <  0.01;  Solution  Option  Generation, 
F=1 1.56,  p  <  0.01;  Building  Common  Ground,  F=20.37,  p  <  0.001;  and  Storyboarding,  F=1 1.90, 
p  <  0.01.  During  Collaborative  Team  Problem  Solving,  face-to-face  teams  spent  more  time 
engaged  in  Individual  Mental  Model  Development,  Knowledge  Interoperability,  Visualization 
and  Representation  of  Meaning,  and  Building  Common  Ground  than  asynchronous-distributed 
teams.  However,  asynchronous-distributed  teams  spent  more  time  engaged  in  Individual  Task 
Knowledge  Development,  Solution  Option  Generation,  and  Storyboarding,  than  face-to-face 
teams.  During  Collaborative  Team  Problem  Solving,  there  was  also  a  significant  difference 
found  between  the  uncertainty  conditions  for  the  subprocess  of  Solution  Option  Generation, 
F=6.45,  p  <  0.05.  Teams  in  the  static  knowledge  condition  spent  more  time  engaged  in  Solution 
Option  Generation  than  teams  in  the  dynamic  knowledge  condition.  The  interaction  across 
collaboration  mode  and  knowledge  uncertainty  for  Solution  Option  Generation  was  also 
significant,  F=6.51  ,  p  <  0  .05.  During  Team  Consensus,  significant  differences  were  only  found 
for  two  subprocesses.  Face-to-face  teams  spent  more  time  engaged  in  both  Knowledge 
Interoperability,  F=13.53,  p  <  0.001,  and  Visualization  and  Representation  of  Meaning,  F=4.40, 
p  <  0.05,  than  asynchronous-distributed  teams.  During  Outcome,  Evaluation,  and  Revision,  there 
were  no  significant  differences  found  for  any  of  the  subprocesses. 

Figure  5  shows  how  the  macrocognitive  major  processes  supported  each  of  the  team 
collaboration  stages,  while  figure  9  shows  how  the  macrocognitive  subprocesses  were  nested 
within  the  macrocognitive  major  processes.  Figure  10  combines  these  results  with  those  just 
reported  related  to  how  the  subprocesses  also  support  the  team  collaboration  stages  to  form  an 
empirically  based  revised  taxonomy  using  the  new  definitions  (Letsky,  et  ah,  2007). 
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Figure  10:  New  Taxonomy  of  the  Collaboration  Stages,  Macrocognitive  Major  Processes,  and 

Subprocesses 
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HOW  TEAMS  TRANSITION  BETWEEN  THE  STAGES  AND  MAJOR  PROCESSES 


Regarding  the  dynamic  nature  of  the  team  collaboration  stages  and  macrocognitive  major 
processes,  figure  1 1  was  taken  from  Warner  and  Wroblewski’s  (2005;  2006)  presentations  of  the 
data  from  the  original  NEO  experiment  (Warner,  Letsky,  and  Cowen,  2005).  It  depicts  the 
probabilities  of  the  transitions  between  the  team  collaboration  stages  across  the  two 
collaboration  conditions. 


Figure  11:  Transition  Probabilities  for  the  Team  Collaboration  Stages 

In  order  to  analyze  the  transition  probabilities  between  the  macrocognitive  major  processes, 
all  the  transitions  between  two  different  macrocognitive  major  processes  were  counted,  totaled, 
and  converted  into  probabilities.  A  2x2  ANOVA  was  used  to  analyze  these  transition 
probabilities  across  collaboration  mode  and  knowledge  uncertainty.  There  was  a  significant  main 
effect  for  collaboration  mode,  F=7.22,  p  <  0.001;  however,  there  was  no  significant  difference 
across  knowledge  uncertainty,  F=1 .81,  p  =  0.16.  The  following  transition  probabilities  between 
the  macrocognitive  major  processes  were  significant:  Individual  Knowledge  Building  — >  Team 
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Knowledge  Building,  F=22.16,  p  <  0.001;  Individual  Knowledge  Building  — >  Developing 
Shared  Problem  Conceptualization,  F=19.45,  p  <  0.001;  Individual  Knowledge  Building  — ■» 
Team  Consensus  Development,  F=588.42,  p  <  0.001;  Team  Knowledge  Building  — >  Individual 
Knowledge  Building,  F=  19.16,  p  <  0.001;  Team  Knowledge  Building  — »  Developing  Shared 
Problem  Conceptualization,  F=4.31  ,  p  <  0  .05;  Team  Knowledge  Building  — >  Team  Consensus 
Development,  F=1 17.59,  p  <  0.001;  Developing  Shared  Problem  Conceptualization  — > 
Individual  Knowledge  Building,  F=24.56,  p  <  0.001;  Developing  Shared  Problem 
Conceptualization  — >  Team  Knowledge  Building,  F=5.97,  p  <  0.05;  Developing  Shared  Problem 
Conceptualization  — >  Team  Consensus  Development,  F=267.94,  p  <  0.001;  Team  Consensus 
Development  — >  Individual  Knowledge  Building,  F=437.09,  p  <  0.001;  Team  Consensus 
Development  — >  Team  Knowledge  Building,  F=  111.01  ,  P  <  0.001;  Team  Consensus 
Development  — >  Developing  Shared  Problem  Conceptualization,  F=145.04,  p  <  0.001;  and  Team 
Consensus  Development  —*■  Outcome  Appraisal,  F=9.91,  p  <  0.01. 

There  were  some  significant  univariate  differences  across  collaboration  mode  (see  figure 
12);  however,  there  were  no  differences  found  across  knowledge  uncertainty.  Face-to-face  teams 
had  a  higher  probability  than  asynchronous-distributed  teams  of  engaging  in  the  following 
macrocognitive  major  process  transitions:  Individual  Knowledge  Building  — >  Team  Knowledge 
Building,  F=10.68,  p  <  0.01;  Team  Knowledge  Building  — >  Individual  Knowledge  Building, 
F=9.20,  p  <  0.01;  Team  Knowledge  Building  — >  Developing  Shared  Problem  Conceptualization, 
F=4.31,  p  <  0.05;  Team  Knowledge  Building  — >  Team  Consensus  Development,  F=  62.04,  p  < 
0.001;  and  Team  Consensus  Development  — >  Team  Knowledge  Building,  F=60.94,  p  <  0.001. 
Asynchronous-distributed  teams  had  a  higher  probability  than  face-to-face  teams  of  engaging  in 
the  following  transitions:  Individual  Knowledge  Building  — >  Developing  Shared  Problem 
Conceptualization,  F=7.77,  p  <  0.01;  Developing  Shared  Problem  Conceptualization  — > 
Individual  Knowledge  Building,  F=6.09,  p  <  0.05;  Developing  Shared  Problem 

Conceptualization  — >  Team  Consensus  Development,  F=47.47,  p  <  0.001;  Team  Consensus 
Development  — >  Developing  Shared  Problem  Conceptualization,  F=23.27,  p  <  0.001;  and  Team 
Consensus  Development  —*■  Outcome  Appraisal,  F=4.61,  p  <  0.05. 
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Figure  12:  Probabilities  of  Transitions  between  the  Major  Macrocognitive  Processes 


DISCUSSION 

The  goal  of  the  current  re-analyses  was  to  examine  the  consistency  and  contributions  of  the 
team  collaboration  stages  and  macrocognitive  processes  proposed  by  Letsky  and  colleagues 
(2007)  in  order  to  facilitate  a  better  understanding  of  the  team  collaboration  process.  The  results 
of  the  re-analyses  support  the  contribution  of  the  four  collaboration  stages  during  the  process  of 
team  collaboration  while  solving  the  NEO  scenario-based  problem.  The  contribution  of  all  live 
major  macrocognitive  processes  during  team  collaboration  was  also  significant.  The  results  of 
the  present  re-analyses  provided  support  for  some  of  the  macrocognitive  subprocesses  in  terms 
of  both  their  consistency  and  contribution.  In  addition,  there  were  significant  differences, 
depending  on  the  collaboration  mode  used  by  the  teams,  in  terms  of  the  collaboration  stages, 
macrocognitive  major  processes,  and  macrocognitive  subprocesses.  In  the  following  sections,  the 
specific  results  regarding  the  team  collaboration  stages,  macrocognitive  major,  and  subprocesses 
will  be  discussed.  The  subsequent  sections  will  discuss  how  teams  transition  between  the  stages 
and  major  processes,  the  consistency  and  contribution  of  the  stages  and  processes,  the  new 
taxonomy,  and  the  evolution  of  the  team  collaboration  model.  Finally,  areas  for  future  research 
will  be  discussed. 
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THE  TEAM  COLLABORATION  STAGES 

The  results  from  the  original  analyses  of  the  NEO  transcripts  and  experimental  data 
(Warner,  Letsky,  and  Cowen,  2005)  regarding  the  team  collaboration  stages  were  that,  overall,  a 
significant  amount  of  time  was  spent  in  each  of  the  four  team  collaboration  stages.  As  was 
mentioned  earlier,  this  result  was  significant  for  the  NEO  scenario,  but  not  for  the  murder 
mystery  scenario  (Warner,  Letsky,  and  Cowen,  2005),  during  which  teams  did  not  spend  a 
significant  amount  of  time  engaged  in  the  Outcome,  Evaluation,  and  Revision  stage.  The 
researchers  attributed  this  finding  to  the  types  of  solutions  required  to  solve  the  two  different 
scenarios.  The  murder  mystery  scenario  had  a  fixed  solution,  whereas  researchers  created  the 
NEO  scenario  (Warner,  Wroblewski,  and  Shuck,  2004)  to  have  numerous  possible  solutions  that 
varied  in  terms  of  how  optimal  they  were  (compared  to  those  planned  by  experts)  and  this 
seemed  to  lead  to  teams  spending  a  significant  amount  of  time  in  the  Outcome,  Evaluation,  and 
Revision  stage.  Warner  and  colleagues  (2005)  concluded  that,  based  on  their  data,  the  team 
collaboration  stages  seem  to  be  dependent  upon  the  type  of  task.  Hutchins  (2008)  also  found 
support  for  the  notion  that  the  team  collaboration  stages  are  domain  dependent. 

There  were  significant  differences  across  collaboration  mode  for  only  two  of  the  team 
collaboration  stages.  The  face-to-face  teams  spent  more  time  in  Knowledge  Construction  than 
the  asynchronous-distributed  teams.  However,  the  asynchronous-distributed  teams  spent  more 
time  in  the  Collaborative  Team  Problem  Solving  stage  than  the  face-to-face  teams.  This  means 
that,  based  on  the  definitions  of  the  stages  (see  Letsky,  et  al.,  2007),  during  face-to-face 
collaboration  teams  spent  more  time  reading,  clarifying  information  and  understanding  the 
problem  than  the  asynchronous-distributed  teams.  Whereas,  the  asynchronous-distributed  teams 
spent  more  time  communicating  data  and  knowledge  to  develop  solutions  to  the  problem  than 
face-to-face  teams.  When  communicating  in  a  face-to-face  situation,  team  members  seemed  to  be 
more  likely  to  overtly  convey  their  need  to  read  infonnation  and  share  their  thoughts  while 
trying  to  understand  the  problem  (e.g.,  through  speech,  which  is  then  coded  during 
communication  analyses  by  raters)  than  when  they  are  communicating  in  an  asynchronous- 
distributed  manner.  Time  spent  reading  information  to  oneself  or  trying  to  understand  the 
problem  without  any  overt  communication  (e.g.,  speaking  or  typing)  was  not  measured  in  this 
study  since  only  the  overt  communication  was  coded  during  the  communication  analysis  for  this 
experiment.  The  second  difference  across  collaboration  mode  was  that  asynchronous-distributed 
teams  seemed  to  spend  more  time  engaged  in  the  Collaborative  Team  Problem  Solving  stage,  in 
other  words,  communicating  (e.g.,  through  typing  messages  back  and  forth)  data  and  knowledge 
to  each  other,  than  face-to-face  teams.  This  may,  in  part,  be  because  face-to-face  teams  were 
better  able  to  quickly  communicate  and  assess  the  need  to  further  discuss  such  information 
through  gestures,  facial  expressions,  or  even  using  words  such  as  “this”  or  “that”  in  reference  to 
people,  places,  or  things  that  may  be  more  easily  confused  when  communicating  in  an 
asynchronous-distributed  manner.  Both  of  these  differences  across  collaboration  mode  need  to 
be  further  examined  in  future  research  in  order  to  be  able  to  gain  a  better  understanding  of  the 
nonverbal  (e.g.,  gestures)  and  internalized  components  of  the  team  collaboration  process.  This 
topic  will  be  discussed  further  in  the  Future  Research  section  of  this  paper. 
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THE  MACROCOGNITIVE  MAJOR  PROCESSES 

The  present  re-analyses  were  based  on  the  most  recent  version  of  the  taxonomy  and 
definitions  (Letsky,  et  al.,  2007).  The  original  team  collaboration  stages  data  from  Warner, 
Letsky,  and  Cowen’s  study  (2005)  was  used,  but  the  re-analyses  extended  the  original  findings  to 
include  how  the  new  macrocognitive  major  processes  support  these  stages.  The  general 
breakdown  of  the  team  collaboration  stages  by  the  macrocognitive  major  processes  is  shown  in 
figure  5. 

Overall,  there  was  a  significant  difference  found  across  collaboration  mode  for  the 
macrocognitive  major  processes  as  they  supported  the  team  collaboration  stages.  During  the 
Knowledge  Construction  stage,  two  of  the  major  processes  differed  in  terms  of  the  percentage  of 
time  spent  based  on  the  collaboration  mode.  During  this  stage,  face-to-face  teams  spent 
significantly  more  time  engaged  in  the  process  of  Team  Knowledge  Building  than  asynchronous- 
distributed  teams.  However,  asynchronous-distributed  teams  spent  more  time  in  the  Developing 
Shared  Problem  Conceptualization  process  than  face-to-face  teams  during  the  Knowledge 
Construction  stage.  During  the  Collaborative  Team  Problem  Solving  stage,  there  were  also  two 
major  processes  that  were  found  to  differ  significantly  across  collaboration  mode.  During  this 
stage,  the  face-to-face  teams  spent  more  time  in  the  Team  Knowledge  Building  process  than  the 
asynchronous-distributed  teams.  Asynchronous-distributed  teams  spent  significantly  more  time 
engaged  in  Developing  Shared  Problem  Conceptualization  than  the  face-to-face  teams. 

These  results  correspond  to  the  overall  differences  found  between  the  macrocognitive  major 
processes  when  they  were  analyzed  independently  of  the  team  collaboration  stages.  Overall, 
face-to-face  teams  spent  more  time  engaged  in  Team  Knowledge  Building  than  asynchronous- 
distributed  teams.  One  possible  explanation  for  this  may  relate  to  the  definition  that  the 
researchers  used  to  code  the  transcripts  during  communication  analyses.  Team  Knowledge 
Building  is  described  as  occurring  when  all  team  members  participate  in  clarifying  information 
(e.g.,  answering  a  question)  to  build  team  knowledge  (see  Letsky,  et  al.,  2007).  While  working  in 
an  asynchronous-distributed  manner,  it  is  possible  that  team  members  may  not  always  type  what 
they  may  be  thinking  while  information  is  being  clarified.  For  example,  in  a  face-to-face 
conversation,  one  person  may  ask  a  question,  a  second  person  on  the  team  may  provide  the 
answer,  and  the  third  person  may  follow  with  a  statement  of  agreement  (e.g.,  okay  or  uh-huh). 
Since  all  three  people  on  the  team  were  involved  in  this  part  of  the  conversation,  it  would  be 
coded  as  Team  Knowledge  Building  by  the  raters  during  communication  analysis.  However,  in 
the  asynchronous-distributed  condition,  the  third  person  on  the  team  may  not  bother  typing 
“okay”,  but  may  be  nodding  (which  would  not  be  identified  through  the  communication 
analyses),  or  simply  may  note  the  newly  clarified  infonnation  to  him/herself.  This  would  not  be 
coded  as  Team  Knowledge  Building  because  only  two  out  of  three  team  members  have  typed 
anything  that  will  be  coded  during  the  communication  analyses.  Future  research  that 
incorporates  other  measures  of  team  members’  behaviors  during  collaboration,  (e.g.,  through 
eye-tracking  and  the  monitoring  of  gestures)  will  facilitate  a  better  understanding  of  this 
difference  between  communication  patterns  between  face-to-face  and  asynchronous-distributed 
teams. 
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The  second  overall  difference  found  between  the  macrocognitive  major  processes  was  that 
asynchronous-distributed  teams  spent  more  time  engaged  in  Developing  Shared  Problem 
Conceptualization  than  face-to-face  teams.  Once  again,  face-to-face  teams  may  have  had  an 
advantage  over  asynchronous-distributed  teams  that  led  to  this  finding.  Developing  Shared 
Problem  Conceptualization  involves  team  members  sharing  their  understanding  of  problem 
goals,  characteristics  of  the  environment,  and  rules  for  operating  the  generation  of  quality 
problem  solutions  (see  Letsky,  et  al.,  2007).  People  who  need  to  communicate  from  different 
locations  may  feel  the  need  to  spend  a  certain  amount  of  time  making  sure  that  each  person 
involved  has  come  to  the  same  understanding  concerning  the  team’s  goals  and  rules  or 
guidelines  of  the  problem.  Face-to-face  teams  may  not  need  to  spend  as  much  time  overtly 
making  sure  of  this  type  of  team-shared  understanding  since  they  are  able  to  see  whether  or  not 
each  team  member  has  finished  reading  something,  has  seen  a  particular  map,  or  even  if 
someone  has  a  confused  look  on  their  face. 

Gutwin  and  Greenberg  (2004)  discuss  the  importance  of  awareness  during  distributed 
collaboration,  specifically  an  awareness  of  “what  is  going  on”  (Endsley,  1995  as  cited  in  Gutwin 
and  Greenberg,  2004).  During  the  process  of  Developing  Shared  Problem  Conceptualization, 
team  members  communicate  about  “what  is  going  on”  in  terms  of  important  guidelines,  goals, 
and  characteristics  of  the  environment.  Gutwin  and  Greenberg  (2004)  suggest  that  it  is  not  only 
verbal  communication  that  is  important  during  team  collaboration,  but  also: 

“team  collaboration  includes  activities  such  as  using  environmental  cues  to  establish  a  common  ground  of 
understanding,  seeing  who  is  around  and  what  they  are  doing,  monitoring  the  state  of  artifacts  in  a  shared 
work  setting,  noticing  other  people’s  gestures  and  what  they  are  referring  to,  and  so  on”  (2004,  p.  177). 

In  their  chapter,  the  authors  discuss  the  fact  that  distributed  communication  systems  and 
tools  often  do  not  offer  a  lot  of  support  regarding  maintaining  awareness,  such  as  shared  white¬ 
boards  (Gutwin  and  Greenberg,  2004),  which  were  not  a  part  of  the  collaboration  environment 
used  for  the  NEO  experiment  (see  Warner,  Letsky,  and  Cowen,  2005).  Thus,  it  is  not  surprising 
that,  in  order  to  compensate,  teams  in  the  asynchronous-distributed  condition  spent  more  time 
overtly  communicating  information  to  improve  team  awareness  compared  to  face-to-face  teams. 

In  terms  of  the  other  macrocognitive  major  processes,  Individual  Knowledge  Building, 
Team  Consensus  Development,  and  Outcome  Appraisal,  no  significant  differences  were  found 
across  collaboration  mode  overall  or  when  examined  in  terms  of  the  team  collaboration  stages 
they  support.  During  both  the  face-to-face  and  asynchronous-distributed  conditions,  teams  spent 
a  significant  amount  of  time  in  each  of  the  macrocognitive  major  processes.  There  were  no 
significant  differences  found  across  the  static  versus  dynamic  knowledge  uncertainty  conditions 
for  any  of  the  five  major  processes,  and  there  were  no  significant  interaction  effects  for  the 
macrocognitive  processes  when  analyzed  individually,  or  in  relation  to  the  team  collaboration 
stages. 
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THE  MACROCOGNITIVE  SUBPROCESSES 

Overall,  during  collaboration  on  the  NEO  problem,  the  present  re-analyses  revealed  that 
teams  spent  a  significant  percentage  of  their  time  engaged  in  fourteen  of  the  macrocognitive 
subprocesses  (see  figure  9).  The  following  subprocesses,  however,  were  not  significant:  Team 
Pattern  Recognition  and  Trend  Analysis,  Team  Mental  Model  Development,  Recognition  of 
Expertise,  Uncertainty  Resolution,  Intuitive  Decision  Making,  Team  Negotiation  of  Solution 
Alternatives,  Replanning,  and  Mental  Simulation. 

In  comparison  to  the  original  analyses  performed  using  an  earlier  taxonomy  and  set  of 
definitions  (Warner,  Letsky,  and  Cowen,  2005),  there  were  several  significant  differences. 
Warner  and  colleagues  (2005)  found  Team  Pattern  Recognition,  Convergence  of  Mental  Models 
(re-named  Team  Mental  Model  Development  in  the  revised  set  of  definitions,  see  Letsky,  et  al., 
2007),  and  Team  Negotiation  of  Solution  Alternatives  to  be  significant,  however,  this  was  not 
the  case  in  the  present  re-analyses.  There  are  several  possible  explanations  for  these  results.  The 
first  possibility  is  that  when  the  two  new  raters  were  coding  the  transcripts,  one  of  the  lines 
during  a  piece  of  the  conversation  was  not  interpreted  to  be  closely  enough  related  and  was, 
therefore,  labeled  separately.  Since,  for  each  of  these  three  subprocesses,  the  team  as  a  whole 
needs  to  take  part  (see  appendix  K),  a  difference  in  the  interpretation  of  one  line  may  have  lead 
to  a  different  label  of  the  other  lines  as  well.  Specifically,  it  may  have  led  the  raters  to  code  the 
lines  as  a  different  subprocess  that  does  not  involve  all  team  members  (e.g.,  Individual  Mental 
Model  Development).  Another  possibility  in  regards  to  the  difference  in  the  results  concerning 
the  Team  Negotiation  of  Solution  Alternatives  subprocess  is  that  the  addition  of  several  new 
subprocesses  in  the  new  taxonomy  led  to  different  labeling  by  the  two  raters.  In  the  new 
taxonomy,  two  of  the  new  subprocesses  were  Feedback  Interpretation  and  Replanning  (see 
Letsky,  et  al.,  2007).  Team  Negotiation  of  Solution  Alternatives,  Feedback  Interpretation,  and 
Replanning  all  have  to  do  with  discussing  solution  alternatives.  In  fact,  in  the  present  re¬ 
analyses,  although  Team  Negotiation  of  Solution  Alternatives  was  not  significant,  Feedback 
Interpretation  was.  Thus,  it  is  possible  that  the  inclusion  of  the  two  new  subprocesses  at  least 
partially  contributed  to  the  different  results.  This  may  have  been  because  there  was  overlap 
between  the  three  subprocesses  and  that  their  definitions  are  not  highly  orthogonal,  or  because 
some  parts  of  the  transcript  that  were  originally  coded  as  Team  Negotiation  of  Solution 
Alternatives  now  fit  better  under  either  the  Feedback  Interpretation  or  Replanning  subprocesses. 

Another  difference  found  between  the  present  re-analyses  and  the  original  analyses 
performed  by  Warner,  Letsky,  and  Cowen  (2005)  was  that  several  of  the  new  subprocesses  that 
were  added  into  the  revised  taxonomy  (see  Letsky,  et  al.,  2007)  were  significant.  The  following 
new  subprocesses  were  significant:  Feedback  Interpretation  (which  was  already  discussed), 
Sharing  Unique  Knowledge,  Knowledge  Sharing,  Knowledge  Transfer,  Critical  Thinking, 
Building  Common  Ground,  and  Storyboarding.  Thus,  it  appears  that  teams  engaged  in  these  new 
subprocesses  often  enough  for  them  to  be  added  into  a  new  model  of  team  collaboration. 

Overall,  there  was  a  significant  difference  found  across  collaboration  mode  for  a  number  of 
the  macrocognitive  subprocesses.  Face-to-face  teams  spent  more  time  than  asynchronous- 
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distributed  teams  engaged  in  the  following  subprocesses:  Knowledge  Interoperability, 
Visualization  and  Representation  of  Meaning,  and  Building  Common  Ground.  During  the 
asynchronous-distributed  condition,  however,  teams  spent  more  time  engaged  in  the  Individual 
Task  Knowledge  Development,  Solution  Option  Generation,  and  Storyboarding  subprocesses. 

In  order  to  discuss  possible  explanations  for  these  differences  across  the  two  collaboration 
modes,  each  macrocognitive  subprocess  needs  to  be  considered  separately.  First,  one  possible 
explanation  for  the  greater  percentage  of  time  spent  by  face-to-face  teams  engaging  in 
Knowledge  Interoperability  relates  to  the  definition  used  during  the  raters’  coding  process. 
Knowledge  Interoperability  refers  to  when  all  team  members  are  exchanging  their  knowledge  of 
the  problem  situation  in  order  to  reach  agreement  in  regards  to  a  topic’s  meaning  (see  Letsky,  et 
ah,  2007).  Similar  to  the  explanation  for  the  macrocognitive  major  process  of  Team  Knowledge 
Building  (which  also  requires  all  team  members  to  participate),  it  may  be  that  the  lack  of  overt 
participation  from  one  team  member  led  the  two  raters  to  label  this  part  of  the  conversation  a 
different  subprocess.  This  would  have  led  to  fewer  lines  being  coded  as  Knowledge 
Interoperability  for  the  asynchronous-distributed  teams.  Another  possible  explanation  for  the 
collaboration  mode  difference  for  the  Knowledge  Interoperability  subprocess  could  be  that  teams 
working  in  a  face-to-face  manner  simply  spent  more  time  engaged  in  the  process  of  exchanging 
their  knowledge  of  the  problem  situation  while  trying  to  come  to  a  consensus  about  a  topic’s 
meaning  than  teams  working  from  different  locations.  Future  research  that  includes  eye-tracking 
measures  or  that  examines  gestures  may  help  to  provide  insight  regarding  differences  in 
Knowledge  Interoperability. 

The  description  provided  by  Letsky  and  colleagues  (2007)  of  the  subprocess  of 
Visualization  and  Representation  of  Meaning  may  help  with  a  better  understanding  of  the 
differences  found  across  the  two  collaboration  modes.  Visualization  of  Meaning  refers  to  when 
team  members  use  visual  aides,  such  as  pictures  or  maps,  to  help  transfer  meaning  to  another 
team  member.  Representation  of  Meaning  refers  to  when  individual  team  members  use  methods 
such  as  note  pads  to  sort  information  into  groups.  Since  face-to-face  teams  each  had  the  same 
maps  and  pictures  in  front  of  them  during  the  experiment,  it  was  easy  for  team  members  to  refer 
others  to  these  visual  aides.  In  addition,  the  face-to-face  teams  could  pass  notes  made  on  a  pad. 
During  the  asynchronous-distributed  teams’  collaboration,  team  members  may  have  spent  time 
referring  to  these  types  of  visual  aides,  however;  they  may  not  have  typed  anything  to  other  team 
members  about  it.  Therefore,  it  would  not  have  ended  up  in  the  transcripts  for  raters  to  code. 
This  is  one  highly  plausible  explanation  for  the  finding  that  a  significantly  higher  percentage  of 
time  was  coded  as  Visualization  and  Representation  of  Meaning  for  the  face-to-face  teams  when 
compared  to  the  asynchronous-distributed  teams.  Warner,  Letsky,  and  Cowen  (2005)  also  found 
that  face-to-face  teams  spent  significantly  more  time  engaged  in  Visualization  and 
Representation  of  Meaning  than  asynchronous-distributed  teams  as  well. 

As  for  the  face-to-face  teams’  spending  more  time  than  asynchronous-distributed  teams 
engaged  in  Building  Common  Ground,  the  explanation  may  lie  in  how  this  secondary  measure 
was  calculated  for  this  study  (see  appendix  K).  For  the  present  re-analyses,  the  researchers 
created  a  composite  Building  Common  Ground  score  that  included  the  lines  coded  as 
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Recognition  of  Expertise,  Team  Pattern  Recognition  and  Trend  Analysis,  Team  Mental  Model 
Development,  Team  Shared  Understanding,  Team  Negotiation  of  Solution  Alternatives,  and 
Feedback  Interpretation.  With  the  exception  of  Recognition  of  Expertise,  all  of  the  other 
subprocesses  listed  above  must  include  all  three  team  members.  Therefore,  the  lack  of  an  overt 
response  by  only  one  team  member  in  the  asynchronous-distributed  condition  may  have  led  to 
fewer  lines  of  conversation  being  coded  as  one  the  above  subprocesses,  as  was  also  likely  the 
case  with  Knowledge  Interoperability. 

For  the  asynchronous-distributed  teams,  a  larger  percentage  of  time  was  spent  engaged  in 
the  Individual  Task  Knowledge  Development  subprocess  than  during  face-to-face  team 
collaboration.  This  macrocognitive  subprocess  involves  question  asking  and  answering  about 
data,  infonnation  about  the  task,  or  planned  actions  (see  appendix  K).  One  possible  reason  for 
this  finding  is  that  the  environment  used  by  the  three-member  asynchronous-distributed  teams 
could  easily  be  used  like  a  chat  room  on  the  internet  or  an  Instant  Messaging  program. 
Therefore,  this  type  of  basic  “Q  and  A”  that  fits  into  the  Individual  Task  Knowledge 
Development  subprocess  was  very  well  facilitated,  and  team  members  used  it  a  great  deal.  A 
second  possible  explanation  is  that  during  face-to-face  collaboration  team  members  may  have 
been  more  likely  to  add  further  insight  or  provide  analysis  in  relation  to  the  answer  to  a  question 
in  keeping  with  the  natural  flow  of  the  conversation.  This  may  have  led  raters  to  select  other 
subprocess  codes  during  communication  analyses  (e.g.,  Iterative  Information  Collection, 
Knowledge  Interoperability,  or  Team  Mental  Model  Development;  see  appendix  K).  Another 
possibility  is  since  they  were  collaborating  from  different  locations,  asynchronous-distributed 
teams  simply  needed  to  spend  additional  time  clarifying  data  or  infonnation  about  the  task.  This 
may  be  (at  least)  partially  because  face-to-face  teams  were  better  able  to  quickly  assess  their 
team  members’  shared  understanding  of  the  data  through  nonverbal  cues  (e.g.,  facial  expression 
or  gestures).  This  was  also  offered  as  one  explanation  for  a  similar  finding  related  to  the 
Collaborative  Team  Problem  Solving  stage  (discussed  earlier).  Finally,  another  possible 
explanation  for  the  Individual  Task  Knowledge  Development  difference  across  collaboration 
mode  is  that  asynchronous— distributed  team  members  may  have  felt  a  greater  need  to  clarify 
data  and/or  information,  perhaps  in  an  attempt  to  “bridge-the-gap”  between  themselves  and  the 
other  team  members. 

Another  macrocognitive  subprocess  for  which  there  were  significant  differences  across 
collaboration  modes  was  Storyboarding.  Asynchronous-distributed  teams  spent  significantly 
more  time  engaged  in  the  Storyboarding  subprocess  than  face-to-face  teams.  The  most  plausible 
explanation  for  this  is  that  the  asynchronous-distributed  environment  that  was  used  facilitated  a 
“storyboarding”-like  format  by  having  team  members  create  “cards”  and  post  them  so  that  other 
team  members  could  read  whatever  was  on  them.  This  seems  to  have  lead  to  asynchronous- 
distributed  teams  engaging  in  more  Storyboarding  than  the  face-to-face  teams. 
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For  the  Solution  Option  Generation  subprocess  there  was  not  only  a  significant  difference 
found  across  collaboration  inodes,  but  across  knowledge  uncertainty  conditions  as  well. 
Asynchronous-distributed  teams  spent  more  time  engaged  in  the  Solution  Option  Generation 
subprocess  than  teams  in  the  face-to-face  condition.  Teams  in  the  static  knowledge  condition 
spent  more  time  in  the  Solution  Option  Generation  subprocess  than  team  in  the  dynamic 
knowledge  condition.  The  interaction  was  also  significant. 

The  fact  that  teams  in  the  asynchronous-distributed  condition  spent  more  time  engaged  in 
Solution  Option  Generation  may  be  due  to  the  nature  of  the  environment  used  for  this  condition. 
As  was  just  discussed  with  the  Storyboarding  subprocess,  the  program  used  by  the 
asynchronous-distributed  teams  was  set  up  to  have  individual  team  members  post  electronic 
“cards”  with  typed  information  on  them  for  other  team  members  to  see.  Thus,  it  facilitated  the 
brainstorming  of  ideas  about  approaches  to  the  final  solution,  which  by  definition,  is  a  part  of  the 
Storyboarding  subprocess  (see  appendix  K). 

As  for  the  difference  across  knowledge  uncertainty  during  Solution  Option  Generation,  a 
potential  explanation  for  this  finding  may  lie  within  the  definitions  of  several  other  subprocesses 
and  the  nature  of  the  manipulation  in  the  dynamic  knowledge  condition.  During  the  dynamic 
knowledge  condition,  selected  information  (e.g.,  location  of  the  rebel  forces  and  the  weather) 
was  changed  through  an  update  provided  by  the  experimenter  at  a  standard  time  during  the 
experiment.  This  meant  that  for  these  teams  in  the  dynamic  knowledge  condition,  the  new 
information  cause  changes  to  their  original  plan.  In  other  words,  the  solution  options  that  they 
may  have  already  generated  may  need  to  be  changed  based  on  the  newly  changed  infonnation. 
Once  a  team  began  to  make  changes  to  their  original  plan,  the  raters  needed  to  look  at  the 
definitions  of  several  other  subprocesses  that  relate  to  changes  made  to  the  original  plan, 
including  Team  Negotiation  of  Solution  Alternatives  and  Replanning  (see  appendix  K).  While 
brainstorming  and  initially  coming  up  with  ideas  for  the  final  plan  were  coded  as  the  Solution 
Option  Generation  subprocess,  changes  made  to  these  solution  options  were  coded  as  either  the 
Team  Negotiation  of  Solution  Alternatives  or  Replanning  subprocesses.  By  design,  the  nature  of 
the  dynamic  knowledge  condition  often  caused  teams  to  need  to  make  changes  to  their  initial 
plan.  Therefore,  a  lower  percentage  of  time  ended  up  coded  as  Solution  Option  Generation 
because  the  ideas  and  suggestions  generated  as  possible  solution  options  after  the  experimenter 
updated  the  infonnation  differed  from  the  initial  plan  and,  therefore,  needed  to  be  coded  as  one 
of  several  other  subprocesses. 

The  macrocognitive  subprocesses  were  also  examined  in  terms  of  how  they  were  nested 
within  each  of  the  macrocognitive  major-processes.  These  results  are  shown  in  figure  9. 
Examination  of  the  differences  across  collaboration  mode  and  knowledge  uncertainty  for  the 
subprocesses  nested  within  the  major  macrocognitive  processes  revealed  some  significant 
differences.  Within  Individual  Knowledge  Building,  face-to-face  teams  spent  a  larger  percentage 
of  their  time  engaged  in  the  subprocesses  of  Knowledge  Interoperability,  Visualization  and 
Representation  of  Meaning,  and  Building  Common  Ground  than  asynchronous-distributed 
teams.  Asynchronous-distributed  teams,  however,  spent  more  time  engaged  in  Knowledge 
Sharing  during  Individual  Knowledge  Building  than  face-to-face  teams.  When  compared  to  the 
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overall  results  for  the  occurrences  of  these  subprocesses,  Knowledge  Interoperability, 
Visualization  and  Representation  of  Meaning,  and  Building  Common  Ground  were  each,  once 
again,  both  found  to  occur  at  a  higher  rate  in  the  face-to-face  teams  than  the  asynchronous- 
distributed  teams.  A  new  difference  across  collaboration  mode  that  emerged  when  examining  the 
nesting  within  Individual  Knowledge  Building;  however,  was  that  asynchronous-distributed 
teams  spent  more  time  in  the  Knowledge  Sharing  subprocess. 

During  Individual  Knowledge  Building,  when  team  members  ask  for  clarification  of  data  or 
information,  or  respond  to  a  clarification  that  was  requested  by  other  team  members  (see 
appendix  J),  the  fact  that  asynchronous-distributed  teams  spent  a  higher  percentage  of  time  (over 
face-to-face  teams)  passing  pieces  of  infonnation  to  another  team  member  (a.k.a.,  Knowledge 
Sharing,  see  appendix  K)  seems  very  reasonable.  In  fact,  in  their  discussion,  Warner,  Letsky,  and 
Cowen  (2005)  summarized  their  original  results  regarding  the  differences  between 
asynchronous-distributed  and  face-to-face  communication  patterns.  They  stated  that 
asynchronous-distributed  teams  seemed  to  focus  more  than  face-to-face  teams  on  individual 
convergence  of  data  to  knowledge  in  order  to  build  knowledge  and  sharing  knowledge  (see 
Warner,  Letsky,  and  Cowen,  2005).  The  current  re-analyses  also  seems  to  support  this  pattern  of 
communication  for  asynchronous-distributed  teams. 

For  the  macrocognitive  major  process  of  Team  Knowledge  Building,  there  were  significant 
differences  across  collaboration  mode  as  well.  While  engaging  in  Team  Knowledge  Building, 
face-to-face  teams  spent  more  time  than  asynchronous-distributed  teams  in  the  following 
subprocesses:  Individual  Task  Knowledge  Development,  Individual  Mental  Model 
Development,  Knowledge  Interoperability,  Visualization  and  Representation  of  Meaning, 
Knowledge  Sharing,  Team  Shared  Understanding,  and  Building  Common  Ground.  These  results 
are  not  surprising  since  overall,  face-to-face  teams  spent  a  significantly  larger  percentage  of  time 
than  asynchronous-distributed  teams  engaged  in  Team  Knowledge  Building.  Team  Knowledge 
Building  is  the  major  process  under  which  all  team  members  participate  in  clarifying  information 
in  order  to  build  team  knowledge  (see  appendix  J).  Once  again,  the  present  re-analyses  provide 
further  support  for  the  communication  patterns  found  by  Warner,  Letsky,  and  Cowen  (2005): 
asynchronous-distributed  teams  spent  more  time  than  face-to-face  teams  on  the  individual 
convergence  of  data  to  knowledge  in  order  to  build  team  knowledge. 

During  Team  Knowledge  Building,  one  subprocess  was  found  to  occur  in  the  asynchronous- 
distributed  condition  more  often  than  the  face-to-face  condition.  The  Storyboarding  subprocess 
during  Team  Knowledge  Building  occurred  more  frequently  during  asynchronous-distributed 
team  collaboration  than  during  face-to-face.  This  finding  is  reflective  of  the  overall  results  for 
this  subprocess,  for  which  overall,  Storyboarding  occurred  more  often  in  the  asynchronous- 
distributed  teams  than  in  the  face-to-face.  Once  again,  it  is  likely  the  type  of  environment  used 
during  the  experiment  to  make  asynchronous-distributed  collaboration  possible  made 
participants  more  likely  to  engage  in  “storyboarding”-like  behaviors. 
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Within  Developing  Shared  Problem  Conceptualization,  there  were  only  two  significant 
differences  across  collaboration  mode.  During  Developing  Shared  Problem  Conceptualization, 
asynchronous-distributed  teams  spent  more  time  engaged  in  Individual  Task  Knowledge 
Development  than  face-to-face  teams.  This  is  not  surprising,  given  the  results  of  the  overall 
analyses  of  this  subprocess.  Overall,  asynchronous-distributed  teams  spent  a  larger  percentage  of 
time  engaged  in  the  Individual  Task  Knowledge  Development  subprocess  than  face-to-face 
teams.  The  results  of  the  analyses  of  Individual  Task  Knowledge  Development  within  the 
Developing  Shared  Problem  Conceptualization  major  process  simply  reflect  the  finding  of  the 
more  general  results.  During  Developing  Shared  Problem  Conceptualization,  the  other 
subprocess  for  which  there  was  a  significant  difference  across  collaboration  was  Storyboarding. 
Once  again,  this  result  reflects  the  overall  findings  for  this  subprocess.  Asynchronous-distributed 
teams  engaged  in  the  Storyboarding  subprocess  more  often  than  face-to-face  teams.  Finally, 
there  were  no  significant  differences  during  Developing  Shared  Problem  Conceptualization 
across  knowledge  uncertainty. 

The  last  major  macrocognitive  process,  under  which  there  were  significant  differences 
across  either  collaboration  mode  or  knowledge  uncertainty  for  the  subprocesses  nested  within, 
was  Team  Consensus  Development.  During  Team  Consensus  Development,  asynchronous- 
distributed  teams  spent  more  time  than  face-to-face  teams  engaged  in  Iterative  Information 
Collection,  Individual  Task  Knowledge  Development,  Solution  Option  Generation,  and 
Storyboarding.  All  four  of  the  findings  for  these  subprocesses  nested  within  the  Team  Consensus 
Development  major  process  reflect  the  general  results  for  the  same  subprocesses  and  are, 
therefore,  not  surprising.  Within  the  Team  Consensus  Development  major  process,  face-to-face 
teams  spent  more  time  than  asynchronous-distributed  teams  engaged  in  the  following 
subprocesses:  Knowledge  Interoperability,  Visualization  and  Representation  of  Meaning,  and 
Building  Common  Ground.  Again,  these  nested  results,  not  surprisingly,  reflect  the  overall 
findings  for  each  of  these  subprocesses.  Finally,  as  was  found  in  the  overall  analyses  for  the 
Solution  Option  Generation  subprocess,  the  analyses  of  this  process  during  Team  Consensus 
Development  revealed  that  the  teams  in  the  static  knowledge  condition  spent  more  time  in 
Solution  Option  Generation  than  teams  in  the  dynamic  knowledge  condition. 

In  order  to  complete  a  new  taxonomy  based  on  the  revised  definitions  proposed  by  Letsky 
and  colleagues  (2007),  the  macrocognitive  subprocesses  were  also  examined  in  relation  to  the 
team  collaboration  stages.  The  overall  results  regarding  the  new  taxonomy  of  the  collaboration 
stages,  macrocognitive  major  processes  and  subprocesses  are  shown  in  figure  10.  There  were 
also  some  significant  findings  regarding  differences  across  collaboration  mode  and  knowledge 
uncertainty  within  the  subprocesses  when  they  were  analyzed  nested  under  the  team 
collaboration  stages  that  they  support.  During  Knowledge  Construction,  face-to-face  teams  spent 
more  time  than  asynchronous-distributed  teams  engaged  in  the  following  subprocesses: 
Knowledge  Interoperability,  Visualization  and  Representation  of  Meaning,  Knowledge  Transfer, 
and  Team  Shared  Understanding.  These  results  are  not  surprising  for  two  reasons.  The  first 
reason  is  that  the  overall  results  when  the  collaboration  stages  were  analyzed  independently  from 
the  subprocesses  that  support  them  were  that  face-to-face  teams  spent  more  time  in  the 
Knowledge  Construction  stage  than  the  asynchronous-distributed  teams.  The  second  reason  is 
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that  the  analyses  for  the  subprocesses  independent  of  the  team  collaboration  stages  revealed  that, 
for  both  Knowledge  Interoperability  and  Visualization  and  Representation  of  Meaning,  a  larger 
percentage  of  time  was  spent  in  these  subprocesses  by  face-to-face  teams  than  teams  in  the 
asynchronous-distributed  condition. 

During  the  Collaborative  Team  Problem  Solving  stage,  face-to-face  teams  spent  more  time 
engaged  in  Individual  Mental  Model  Development,  Knowledge  Interoperability,  Visualization 
and  Representation  of  Meaning,  and  Building  Common  Ground  than  asynchronous-distributed 
teams.  One  explanation  for  these  results  is  that  they  reflect  the  overall  findings  for  each  of  these 
subprocesses  when  analyzed  independently  of  the  team  collaboration  stages  that  they  support. 
However,  asynchronous-distributed  teams  spent  more  time  engaged  in  Individual  Task 
Knowledge  Development,  Solution  Option  Generation,  and  Storyboarding,  than  face-to-face 
teams  during  the  Collaborative  Team  Problem  Solving  stage.  These  differences  between  the  two 
collaboration  modes  seem  to  reflect  the  overall  results  of  the  team  collaboration  stages  when 
analyzed  independently  from  the  macrocognitive  subprocesses.  Overall,  the  asynchronous- 
distributed  teams  spent  significantly  more  time  in  the  Collaborative  Team  Problem  Solving  stage 
than  the  teams  in  the  face-to-face  condition.  Therefore,  the  fact  that  three  of  the  subprocesses 
that  support  this  stage  reflect  this  generalized  finding  seems  very  logical.  During  Collaborative 
Team  Problem  Solving,  there  was  also  a  significant  difference  found  between  the  uncertainty 
conditions  for  the  subprocess  of  Solution  Option  Generation.  Teams  in  the  static  knowledge 
condition  spent  more  time  engaged  in  Solution  Option  Generation  than  teams  in  the  dynamic 
knowledge  condition.  Once  again,  this  finding  reflects  the  overall  results,  independent  from  the 
collaboration  stages,  for  this  subprocess,  and  was  therefore,  not  surprising. 

During  the  Team  Consensus  stage,  significant  differences  were  only  found  for  two 
subprocesses.  Face-to-face  teams  spent  more  time  than  asynchronous-distributed  teams  engaged 
in  both  Knowledge  Interoperability  and  Visualization  and  Representation  of  Meaning.  Once 
again,  both  of  these  subprocesses  were  found  to  more  commonly  occur  during  face-to-face  team 
collaboration  than  asynchronous-distributed  collaboration  when  analyzed  independently  from  the 
team  collaboration  stages.  Within  the  Team  Consensus  stage,  there  were  no  significant 
differences  across  the  knowledge  uncertainty  conditions.  Finally,  during  the  Outcome, 
Evaluation,  and  Revision  stage,  there  were  no  significant  differences  found  for  any  of  the 
subprocesses  across  either  collaboration  mode  or  knowledge  uncertainty. 

HOW  TEAMS  TRANSITION  BETWEEN  THE  TEAM  COLLABORATION  STAGES 

As  was  mentioned  in  the  Introduction  section  of  this  report,  both  the  team  collaboration 
stages  and  macrocognitive  processes  are  dynamic  in  nature.  The  results  of  the  analyses  of  the 
transition  probabilities  between  the  team  collaboration  stages  revealed  that  there  were  no 
significant  differences  across  knowledge  uncertainty;  however,  there  were  differences  across 
collaboration  modes.  The  transition  probabilities  for  the  team  collaboration  stages  are  shown  in 
figure  11  (see  also,  Warner  and  Wroblewski,  2005;  2006).  There  were  significant  differences 
across  collaboration  mode  for  the  following  stage-to-stage  transitions:  from  Knowledge 
Construction  to  Collaborative  Team  Problem  Solving,  from  Collaborative  Team  Problem 
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Solving  to  Knowledge  Construction,  from  Team  Consensus  to  Knowledge  Construction,  from 
Team  Consensus  to  Collaborative  Team  Problem  Solving,  from  Outcome,  Evaluation,  and 
Revision  to  Knowledge  Construction,  from  Outcome,  Evaluation,  and  Revision  to  Collaborative 
Team  Problem  Solving,  and  from  Outcome,  Evaluation,  and  Revision  to  Team  Consensus.  For 
each  of  these  transitions,  the  probability  of  occurrence  was  higher  for  the  asynchronous- 
distributed  teams  than  for  the  face-to-face  teams.  However,  when  you  look  at  the  probabilities 
for  teams  to  stay  within  the  same  stage  between  lines  of  utterances  (e.g.,  one  statement  falls 
under  Knowledge  Construction  and  so  does  the  next  line);  all  of  the  probabilities  are 
significantly  higher  in  the  face-to-face  condition  than  the  asynchronous-distributed  condition. 
The  means  that  face-to-face  teams,  on  average,  were  more  likely  to  have  a  number  of 
consecutive  lines  of  text  in  the  transcripts  coded  as  the  same  stage,  whereas  teams  in  the 
asynchronous-distributed  condition  were  more  likely  to  transition  between  the  stages  during  their 
conversations.  The  most  likely  explanation  for  this  finding  is  that  the  conversations  in  face-to- 
face  condition  more  closely  resembled  a  normal  everyday  conversation  involving  three  people 
talking  while  in  the  same  room.  The  conversation  flowed  from  one  topic  to  another  in  a  smooth 
manner  with  few  abrupt  transitions.  In  the  asynchronous-distributed  condition,  however,  this 
type  of  natural  flow  did  not  appear  to  take  place.  Rather,  team  members  communicated  in  brief 
statements,  and  jumped  around,  not  only  from  topic  to  topic,  but  also  from  team  collaboration 
stage  to  collaboration  stage. 

Another  explanation  for  the  seemingly  abrupt  and  frequent  transitions  between  the 
collaboration  stages  may  be  that  teams  in  the  asynchronous-distributed  condition  only  spent  time 
typing  what  they  absolutely  needed  to  ask  or  convey  to  others.  This  was  noted  by  Warner, 
Letsky,  and  Cowen  (2005),  and  they  proposed  that  the  structure  of  the  EWall  environment 
pennitted  more  effective  and  focused  communication.  They  suggested  that  EWall  helped  teams 
to  focus  on  team  problem  solving  behaviors  (e.g.,  as  was  discussed  earlier,  asynchronous- 
distributed  teams  spent  a  significantly  higher  percentage  of  time  in  the  Collaborative  Team 
Problem  Solving  stage  than  the  face-to-face  teams)  and  helped  team  members  to  communicate 
only  the  most  relevant  infonnation  and  knowledge  to  each  other.  This  notion  was  further 
supported  by  the  fact  that  asynchronous-distributed  teams  made  significantly  fewer  utterances  in 
all  four  of  the  team  collaboration  stages  compared  to  the  face-to-face  teams.  The  asynchronous- 
distributed  teams  did  not  differ,  however,  from  the  face-to-face  teams  in  tenns  of  the  amount  of 
time  needed  to  complete  the  task,  or  in  tenns  of  their  perfonnance  (e.g.,  the  quality  of  their 
solution).  This  means  that  they  achieved  the  same  high  quality  solution  in  the  same  amount  of 
time,  but  the  asynchronous-distributed  teams  simply  required  less  communication  than  teams  in 
the  face-to-face  condition.  Once  again,  Warner,  Letsky,  and  Cowen  (2005)  suggested  that  this 
was  because  EWall  facilitated  a  more  effective  mode  of  communication  and  collaboration 
compared  to  the  face-to-face  collaboration  mode. 
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HOW  TEAMS  TRANSITION  BETWEEN  THE  MACROCOGNITIVE  MAJOR  PROCESSES 

For  the  macrocognitive  major  processes,  overall,  there  were  a  number  of  transition 
probabilities  that  occurred  a  significant  amount  of  the  time  (see  figure  12).  There  were  no 
significant  differences  across  knowledge  uncertainty  regarding  transitions  between  the 
macrocognitive  major  processes.  However,  there  were  a  number  of  significant  differences 
between  the  asynchronous-distributed  and  face-to-face  conditions  for  the  major  process  to  major 
process  transition  probabilities.  Specifically,  teams  in  the  face-to-face  condition  were  more 
likely  than  teams  in  the  asynchronous-distributed  condition  to  transition  from:  Individual 
Knowledge  Building  to  Team  Knowledge  Building,  Team  Knowledge  Building  to  Individual 
Knowledge  Building,  Team  Knowledge  Building  to  Team  Consensus  Development,  and  Team 
Consensus  Development  to  Team  Knowledge  Building.  Teams  in  the  asynchronous-distributed 
condition  were  more  likely  than  teams  in  the  face-to-face  condition  to  transition  from:  Individual 
Knowledge  Building  to  Developing  Shared  Problem  Conceptualization,  Developing  Shared 
Problem  Conceptualization  to  Individual  Knowledge  Building,  Developing  Shared  Problem 
Conceptualization  to  Team  Consensus  Development,  and  Team  Consensus  Development  to 
Developing  Shared  Problem  Conceptualization. 

These  results,  when  compared  to  the  overall  macrocognitive  major  process  occurrence  rates 
broken  down  by  collaboration  mode,  seem  very  plausible.  Teams  in  the  face-to-face  condition 
spent  a  larger  percent  of  time  engaged  in  the  Team  Knowledge  Building  major  process  than 
teams  in  the  asynchronous-distributed  condition.  Therefore,  transition  probabilities  both 
beginning  and  ending  in  this  major  process  were  higher  for  the  face-to-face  teams.  Teams  in  the 
asynchronous-distributed  condition  spent  more  time  than  teams  in  the  face-to-face  condition 
engaged  in  the  major  process  of  Developing  Shared  Problem  Conceptualization.  This  reflects  the 
finding  that  asynchronous-distributed  teams  spent  a  larger  percentage  of  time  engaged  in 
Developing  Shared  Problem  Conceptualization  than  face-to-face  teams.  Therefore,  it  seems  very 
reasonable  that,  when  examining  the  transition  probabilities,  all  of  those  that  were  found  to  occur 
more  often  in  the  asynchronous-distributed  condition  than  in  the  face-to-face  condition  either 
began  or  ended  in  Developing  Shared  Problem  Conceptualization.  These  results,  once  again 
support  the  findings  of  Warner,  Letsky,  and  Cowen  (2005)  that  asynchronous-distributed  teams 
tended  to  focus  on  sharing  knowledge,  whereas,  face-to-face  teams  tended  to  employ  many  more 
processes  to  build  knowledge. 

EXAMINING  CONSISTENCY  AND  CONTRIBUTION 

Warner,  Letsky,  and  Cowen’s  (2005)  research  examined  the  consistency  and  contribution  of 
the  team  collaboration  stages  and  macrocognitive  subprocesses  from  earlier  versions  of  the 
taxonomy  and  definitions.  Their  findings  led  to  the  revised  version  of  the  taxonomy  and 
definitions  proposed  by  Letsky  and  colleagues  (2007).  While  there  were  a  number  of  changes  to 
the  macrocognitive  subprocesses,  the  team  collaboration  stages  remained  unchanged.  In 
addition,  the  new  taxonomy  included  the  five  macrocognitive  major  processes. 
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For  the  team  collaboration  stages,  Warner,  Letsky,  and  Cowen  (2005)  concluded  that  there 
were  significant  differences  regarding  the  contribution  of  the  stages  during  problem  solving  that 
were  domain  dependent.  Specifically,  teams  solving  fixed  solution  collaborative  problems  (e.g., 
the  murder  mystery  scenario)  did  not  go  through  the  Outcome,  Evaluation,  and  Revision  stage, 
whereas  teams  that  collaborated  on  problems  where  they  needed  to  develop  optimized  solutions 
(e.g.,  the  NEO  scenario)  spent  a  significant  amount  time  in  each  of  the  four  stages  outlined  in 
their  model  (see  figure  1).  Since  the  present  re-analyses  used  the  transcripts  from  the  same  NEO 
scenario  experiment  that  Warner,  Letsky,  and  Cowen  (2005)  used  in  their  second  experiment,  no 
additional  information  about  the  consistency  of  the  team  collaboration  stages  across  different 
problem  solving  domains  was  obtained.  However,  additional  information  was  obtained  related  to 
how  dependent  the  coding  of  the  team  collaboration  stages  was  upon  the  raters  who  went 
through  the  transcripts.  Originally,  the  interrater  reliability  between  the  two  raters  for  Warner, 
Letsky,  and  Cowen’s  (2005)  study  was  0.80.  For  the  present  re-analyses,  two  new  raters,  using 
the  same  definitions  and  examples  coded  a  subset  of  the  transcripts  for  the  team  collaboration 
stages.  The  interrater  reliability  between  the  two  new  raters  was  also  0.80.  Therefore,  it  seems 
that  different  raters  can  code  a  transcript  for  the  team  collaboration  stages  with  a  high  degree  of 
consistency  and  that  these  stages  are  not  dependent  upon  the  raters  themselves. 

As  for  their  contribution,  there  were  two  major  differences  for  the  team  collaboration  stages 
across  collaboration  mode.  Face-to-face  teams  spent  a  larger  amount  of  time  than  asynchronous- 
distributed  teams  in  the  Knowledge  Construction  stage.  Asynchronous-distributed  teams  spent 
more  time  than  teams  in  the  face-to-face  condition  in  the  Collaborative  Team  Problem  Solving 
stage.  Based  on  the  definitions  provided  by  Letsky  and  colleagues  (2007),  teams  in  the  face-to- 
face  conditions,  therefore,  spent  more  time  reading,  clarifying  information,  and  trying  to 
understand  the  problem.  Asynchronous-distributed  teams  spent  a  larger  percentage  of  their  time 
communicating  and  analyzing  data  and  knowledge  to  develop  solution  options,  developing, 
rationalizing,  and  discussing  alternative  approaches.  These  results  regarding  the  contributions  of 
the  team  collaboration  stages  were  consistent  across  both  of  the  original  experiments  performed 
by  Warner,  Letsky,  and  Cowen  (2005),  and  the  present  re-analyses. 

Since  the  macrocognitive  major  processes  were  new  additions  to  the  revised  taxonomy  (see 
Letsky,  et  ah,  2007),  the  consistency  and  contribution  of  these  processes  across  different 
domains  has  yet  to  be  studied.  However,  the  results  of  the  present  analyses  revealed  that,  overall, 
all  five  processes  were  significant  and  contributed  to  the  problem  solving  process.  The  interrater 
reliability  between  the  two  raters  was  0.82,  which  is  similar  to  the  interrater  reliability  for  the 
team  collaboration  stages.  This  result  seems  promising  in  terms  of  the  orthogonality  of  the 
definitions.  Since  the  interrater  reliability  was  high,  it  seems  as  though  the  two  raters  were  able 
to  distinguish  between  each  of  the  macrocognitive  major  processes  during  the  coding  process. 
One  of  Letsky  and  colleagues’  (2007)  goals  was  to  have  orthogonal  definitions  for  the  processes. 
Although  future  research  will  need  to  examine  further  these  major  processes  and  their 
consistency  across  different  domains,  the  initial  results  of  the  present  research  seem  encouraging. 

Examination  of  the  macrocognitive  major  processes  also  revealed  that  there  were  several 
differences  between  the  two  collaboration  modes.  The  asynchronous-distributed  teams  were 
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significantly  more  likely  than  face-to-face  teams  to  spend  a  greater  percentage  of  the  time 
engaged  in  Developing  Shared  Problem  Conceptualization.  Teams  in  the  face-to-face  conditions, 
however,  spent  significantly  more  time  in  the  Team  Knowledge  Building  process  that 
asynchronous-distributed  teams.  Examination  of  the  transition  probabilities  between  the 
macrocognitive  major  processes  was  a  second  way  in  which  the  contributions  of  the  processes 
were  looked  at.  The  differences  that  were  found  for  all  five  of  the  major  processes  in  general 
were  consistent  with  the  analyses  of  the  transition  probabilities  between  the  processes.  As  was 
discussed  earlier,  all  of  the  transition  probabilities  that  either  began  with  or  ended  with  the  Team 
Knowledge  Building  major  process  were  higher  in  the  face-to-face  condition  than  in  the 
asynchronous-distributed  condition.  The  transition  probabilities  that  either  started  or  finished 
with  the  Developing  Shared  Problem  Conceptualization,  on  the  other  hand,  were  higher  in  the 
asynchronous-distributed  condition  than  in  the  face-to-face  condition. 

Since  the  revised  taxonomy  (see  Letsky,  et  ah,  2007)  included  a  number  of  new 
subprocesses  and  definitions  (and  changed  or  eliminated  several  of  the  old  subprocesses),  a  full 
comparison  was  not  possible  across  all  of  the  subprocesses  between  the  original  Warner,  Letsky, 
and  Cowen  (2005)  experiments  and  the  present  re-analyses.  Some  of  the  subprocesses,  however, 
remained  the  same.  The  following  subprocesses  were  significant  across  both  sets  of  analyses: 
Iterative  Information  Collection,  Individual  Task  Knowledge  Development,  Individual  Mental 
Model  Development,  Knowledge  Interoperability,  Visualization  and  Representation  of  Meaning, 
Team  Shared  Understanding,  and  Solution  Option  Generation.  For  each  of  these  seven 
macrocognitive  subprocesses,  their  contribution  to  team  collaboration  during  problem  solving 
using  the  NEO  scenario  has  shown  consistency,  even  with  changes  to  the  other  subprocesses  in 
the  taxonomy.  In  addition,  the  interrater  reliability  for  the  macrocognitive  processes  calculated 
between  raters  for  Warner,  Letsky,  and  Cowen’s  (2005)  experiment  was  0.57.  The  interrater 
reliability  for  the  subprocesses  for  the  present  re-analyses  was  0.71.  This  significant  increase  in 
the  reliability  between  the  two  raters  may  be  because  the  definitions  are  becoming  more  clear 
and  understandable,  the  additional  examples  that  were  added  were  highly  beneficial  to  the  raters, 
and  the  revised  set  of  definitions  are  more  orthogonal  than  the  original  set  (see  Letsky,  et  ah, 
2007;  Warner,  Letsky,  and  Cowen,  2005). 

For  three  of  the  subprocesses,  the  issue  of  orthogonality  of  the  definitions  was  already 
raised  earlier  in  the  Discussion  section.  In  Warner,  Letsky,  and  Cowen’s  (2005)  study,  the 
macrocognitive  process  of  Team  Negotiation  of  Solution  Alternatives  was  significant.  In  the 
revised  taxonomy,  however,  two  subprocesses  were  added  that  deal  with  assessing  and/or 
changing  solution  options:  Replanning  and  Feedback  Interpretation  (see  Letsky,  et  ah,  2007). 
The  addition  of  these  subprocesses  with  highly  related  definitions  may  have  led  to  raters 
selecting  one  of  the  other  two  subprocesses  during  coding  in  some  instances.  This  would  explain 
why  Team  Negotiation  of  Solution  Alternatives  was  not  significant,  but  Feedback  Interpretation 
was  significant  in  the  present  re-analyses.  Future  research  will  need  to  examine  this  issue  further. 

In  addition  to  the  subprocesses  already  discussed,  six  more  subprocesses  were  significant: 
Sharing  Unique  Knowledge,  Knowledge  Sharing,  Knowledge  Transfer,  Critical  Thinking, 
Building  Common  Ground,  and  Storyboarding.  Along  with  Feedback  Interpretation,  these  seven 
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subprocesses  were  found  to  contribute  a  significant  percentage  of  the  time  in  supporting  the  team 
collaboration  stages  during  the  team  problem  solving  process.  However,  unlike  Iterative 
Information  Collection,  Individual  Task  Knowledge  Development,  Individual  Mental  Model 
Development,  Knowledge  Interoperability,  Visualization  and  Representation  of  Meaning,  Team 
Shared  Understanding,  and  Solution  Option  Generation,  there  are  only  the  results  from  the 
present  re-analyses  that  provide  support  in  regards  to  their  significant  contributions  to  team 
collaboration.  Additional  research  will  be  needed  in  this  area  to  further  provide  support 
concerning  the  contributions  of  these  subprocesses,  and  will  also  need  to  address  their 
consistency  across  different  problem  domains. 

Research  using  different  problem  domains  will  also  provide  insight  as  to  each  of  the  new 
subprocesses  that  did  not  appear  to  contribute  a  significant  percentage  of  the  time  during  team 
collaboration  using  the  NEO  scenario  (e.g.,  Uncertainty  Resolution,  Metal  Simulation,  Intuitive 
Decision  Making,  Replanning,  Recognition  of  Expertise,  etc.).  For  each  of  these  subprocesses,  a 
number  of  questions  still  need  to  be  addressed.  Are  the  definitions  orthogonal  with  the  other 
definitions?  Is  communication  analysis  the  best  way  to  accurately  measure  each  of  the 
subprocesses?  Does  the  contribution  of  these  subprocesses  during  team  collaboration  depend  on 
the  problem  domain?  These  issues  will  be  discussed  further  in  the  Future  Research  section  of 
this  report. 

THE  NEW  TAXONOMY  AND  THE  EVOLUTION  OF  THE  TEAM  COLLABORATION 
MODEL 

The  new  taxonomy  shown  in  figure  10  was  based  upon  the  empirical  results  of  the  present 
re-analyses.  It  summarizes  the  team  collaboration  stages,  macrocognitive  major  processes,  and 
subprocesses  that  had  significant  contributions  to  the  team  collaboration  process.  It  shows  which 
of  the  major  processes  supported  the  collaboration  stages  during  the  team  problem  solving 
process,  and  which  subprocesses  nested  within  each  of  the  major  processes.  However,  the 
taxonomy  does  not  show  how  teams  transition  between  the  stages  and  the  processes  during  team 
collaborative  problem  solving.  The  analyses  that  were  perfonned  on  the  transition  probabilities 
facilitated  a  better  understanding  of  these  relationships.  Combining  the  new  empirically  based 
taxonomy  with  the  overall  significant  transition  probabilities  culminated  in  the  team 
collaboration  model  shown  in  figure  13  (see  also,  Warner,  Biron  and  Burkman,  2008).  In  the 
evolutionary  process,  the  model  shown  here  represents  the  logical  next  phase. 

A  brief  example  was  included  in  a  recent  presentation  of  these  re-analyses  and  the  new 
model  (Warner,  Biron,  and  Burkman,  2008)  in  order  to  help  explain  the  model  and  is  included  in 
appendix  L.  It  includes  a  brief  piece  of  a  conversation,  which  can  be  used  to  describe  the  team 
collaboration  stage,  macrocognitive  major  and  subprocesses  associated  with  the  excerpt.  The 
example  (see  appendix  L)  included  all  three  team  members  talking  (e.g.,  the  weapons  expert,  the 
environmental  expert,  and  the  intelligence  expert).  The  weapon’s  expert  began  by  asking  a 
question  to  build  knowledge  and  try  to  improve  his/her  understanding  of  about  traveling  times 
and  distances.  He  or  she  was  trying  to  clarify  information,  and  while  doing  this,  he/she  referred 
other  team  members  to  a  map.  The  second  person  to  speak  was  the  environmental  expert.  He/she 
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tried  to  provide  some  clarification  and  answer  the  weapons  expert’s  question  without  applying 
the  knowledge  to  a  solution.  In  addition,  while  trying  to  answer  a  teammate’s  question,  he/she 
also  used  the  visual  representation  of  the  previously  referred  to  map.  The  last  person  to  speak  in 
this  example  is  the  intelligence  expert.  He/she  answered  the  weapons  expert’s  original  question 
by  using  the  map  (e.g.,  measuring  and  reading  information  on  the  map)  to  transfer  meaning  and 
provide  clarification  to  other  team  members. 
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Figure  13:  A  New  Team  Collaboration  Model 

When  team  members  are  reading  (e.g.,  information  from  a  map),  clarifying  information  and 
trying  to  understand  the  problem  or  some  aspect  of  it  (e.g.,  traveling  times  or  distances)  as  they 
were  in  this  example,  they  are  engaged  in  the  team  collaboration  stage  known  as  Knowledge 
Construction  (see  Letsky,  et  ah,  2007).  From  this  example,  it  is  also  clear  that  all  three  team 
members  are  participating  in  trying  to  clarify  information  in  order  to  build  team  knowledge. 
They  are  doing  this  through  question  asking  and  answering.  Based  on  the  definitions  for  the  five 
macrocognitive  major  processes  (see  Letsky,  et  ah,  2007;  appendix  J),  this  is  classified  as  Team 
Knowledge  Building.  Finally,  each  of  the  team  members  spends  time  referring  to  the  map.  The 
map  is  a  visual  aid  or  representation,  and  the  team’s  usage  of  it  makes  their  questions  and 
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statement  easy  to  classify  as  the  macrocognitive  subprocess  of  Visualization  and  Representation 
of  Meaning  (for  all  subprocess  definitions  see  appendix  K). 

Figure  14  is  a  visual  representation  of  this  example  based  on  the  new  team  collaboration 
model  shown  in  figure  13.  It  shows  that,  while  working  on  a  team  collaboration  task,  the  team 
members  in  the  example  were  working  within  the  Knowledge  Construction  stage  and 
participating  in  Team  Knowledge  Building.  While  building  knowledge,  the  team  was  using  the 
macrocognitive  subprocess  of  Visualization  and  Representation  of  Meaning.  However,  putting 
forth  the  team  collaboration  model  in  figure  13  and  the  accompanying  example  by  no  means 
exposes  the  complete  nature  of  team  collaboration,  the  stages,  and  the  processes  that  support 
them.  The  example  was  included  to  help  facilitate  a  better  understanding  of  the  model  and  how 
its  components  work  together  to  represent  the  communication  that  took  place  during  the  teams’ 
collaboration  communication  analyses  data  using  the  revised  taxonomy  and  definitions  proposed 
by  Letsky  and  colleagues  (2007),  is  still  a  work  in  progress  in  tenns  of  fully  expounding  the 
team  collaboration  process.  At  the  bottom  of  the  model  there  are  still  macrocognitive 
subprocesses  for  which  little  is  know  to  date  about  their  contribution  to  the  team  collaboration 
process.  Future  research  will  need  to  address  this,  as  well  as  a  number  of  other  aspects  of  the 
team  collaboration  process  before  a  fully  comprehensive  and  predictive  model  can  be  produced. 

FUTURE  RESEARCH 


While  the  new  taxonomy  and  model  of  team  collaboration  give  insight  as  to  how  the  teams 
collaborated  on  the  NEO  scenario,  there  is  still  a  lot  of  research  that  needs  to  be  done  to 
determine  the  consistency  and  contribution  of  the  collaboration  stages,  major  and  subprocesses 
across.  For  example,  the  team  collaboration  stages  have  been  studied  across  different  problem¬ 
solving  domains  in  the  lab  (e.g.,  the  murder  mystery  scenario  and  the  NEO  scenario;  see  Warner, 
Letsky,  and  Cowen,  2005),  outside  the  lab  through  experiments,  and  in  real  world  settings  (e.g., 
the  Maritime  Interdiction  Operations,  the  Air  warfare  decision  making,  and  transcripts  from  the 
Firefighters  from  11  September  2001;  see  Hutchins,  2008).  However,  the  macrocognitive  major 
processes  have  only  been  examined  during  the  NEO  scenario.  In  addition,  while  Warner,  Letsky, 
and  Cowen’ s  (2005)  study  examined  the  contribution  of  subprocesses,  some  new  subprocesses 
were  added  and  others  have  been  removed  from  the  revised  taxonomy  and  some  of  the 
definitions  were  changed  (see  Letsky,  et  al.,  2007).  Therefore,  at  this  point,  only  a  subset  of  the 
subprocesses  have  been  examined  across  different  domains  (see  Hutchins,  2008;  Warner,  Letsky, 
and  Cowen  2005),  and  future  research  needs  to  examine  both  the  major  and  subprocesses  further 
to  really  assess  their  consistency  and  contribution.  Future  research  in  this  area  will  also, 
hopefully,  provide  insight  as  to  the  contributions  of  some  of  the  subprocesses  that  were  not 
significant  in  the  present  re-analyses  of  the  NEO  data  transcripts  (e.g.,  Mental  Simulation, 
Recognition  of  Expertise,  Uncertainty  Resolution,  etc.),  and  subprocesses  for  which  the  degree 
of  orthogonality  with  other  subprocesses  is  uncertain  (e.g.,  Feedback  Interpretation,  Replanning, 
and  Team  Negotiation  of  Solution  Alternatives). 
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Figure  14:  Visual  Representation  of  the  Example  from  Warner,  Biron,  and  Burkman’s 
(2008)  presentation  as  it  relates  to  the  New  Model  of  Team  Collaboration 

Another  area  for  researchers  who  are  interested  in  this  area  to  investigate  is  how  different 
measures  can  be  used  to  examine  the  team  collaboration  stages  and  macrocognitive  processes 
during  problem  solving.  Some  subprocesses,  such  as  Mental  Simulation,  may  not  always  be 
overtly  expressed  during  the  collaboration  process.  This  subprocess  may  be  one  that  takes  place 
more  internally.  Therefore,  communication  analyses  may  not  be  the  most  accurate  measure.  For 
some  of  the  subprocesses  (e.g.,  Mental  Simulation,  Individual  Mental  Model  Development,  and 
Intuitive  Decision  Making)  that  may  occur  at  a  more  internalized  level,  self-report  measures 
maybe  useful. 

Some  additional  measures  that  may  help  to  provide  additional  information  about  processes 
that  contribute  to  the  team  collaboration  process  during  problem  solving.  For  example,  eye¬ 
tracking  measures  and  analyzing  gestures  that  team  members  make  during  collaboration  may 
provide  information  about  attention  and  nonverbal  communication  that  raters  cannot  code  during 
communication  analyses.  Research  that  includes  these  measures  (and  others)  may  also  help  to 
provide  additional  infonnation  about  how  nonovert  responses  may  affect  the  findings  for 
processes  defined  as  including  all  team  members,  and  in  general,  because  having  multiple 
measures  would  be  useful  for  examining  both  contribution  and  constancy  of  the  macrocognitive 
processes  during  team  collaboration. 

Another  major  area  that  still  needs  to  be  addressed  in  future  research  is  team  differences. 
There  may  be  differences  in  how  teams  collaborate  based  on  a  number  of  different  variables.  For 
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example,  although  Warner,  Letsky,  and  Cowen  (2005)  did  not  find  any  perfonnance  differences 
across  either  of  their  two  independent  variables  (e.g.,  knowledge  uncertainty  and  collaboration 
mode),  for  some  problem  domains,  there  may  be  differences  in  how  high  performing  (e.g.,  teams 
that  score  high  on  predetermined  performance  measures)  and  low  performing  teams  collaborate. 
Some  other  variables  that  are  listed  in  Warner,  Letsky,  and  Cowen’s  (2005)  team  collaboration 
model  (see  figure  1)  that  may  affect  how  team  collaborate  include:  unique  roles,  cultural 
diversity,  and  time  pressure.  In  addition,  although  Warner,  Letsky,  and  Cowen  (2005)  did  not 
find  many  significant  differences  across  knowledge  uncertainty,  perhaps  more  differences  might 
exist  across  different  problem  domains  or  through  different  ways  of  manipulating  knowledge 
uncertainty  (e.g.,  more  changes  in  the  data).  Hutchins  (2008)  discusses  differences  across 
different  problem  domains  as  well.  For  example,  she  discussed  team  collaboration  differences 
between  mission  planning  and  mission  execution  scenarios  and  proposed  that  an  additional 
subprocess,  Request  to  Take  Action,  had  a  significant  contribution  during  team  collaboration  that 
involved  mission  execution  (e.g.,  the  transcripts  from  the  Firefighters  from  1 1  September  2001). 
Depending  on  the  problem  domain  and  the  variables  being  studied,  researchers  may  discover 
additional  subprocesses  that  have  significant  contributions  as  well. 

Future  research  should  address  all  of  these  issues  in  order  to  foster  a  better  understanding  of 
how  teams  collaborate  during  different  types  of  problem  solving  tasks.  Specifically,  researchers 
need  to  focus  on  examining  the  contributions  and  consistency  of  the  collaboration  stages  and 
macrocognitive  major  and  subprocesses.  Consequently,  this  should  lead  to  the  development  of  a 
strong  empirically-based  model  of  team  collaboration  and  help  guide  researchers  toward 
developing  the  best  possible  team  collaboration  tools. 
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CONCLUSIONS 

The  goal  of  this  research  was  to  facilitate  a  better  understanding  of  the  process  of  team 
collaboration.  Through  re-analyzing  previously  collected  data  transcripts  using  the  revised 
taxonomy  and  definitions  proposed  by  Letsky  and  colleagues  (2007),  an  examination  of  the 
consistency  and  contributions  of  the  collaboration  stages,  macrocognitive  major  processes,  and 
subprocesses  was  possible.  The  results  of  the  re-analyses  support  the  contribution  of  the  four 
collaboration  stages  during  the  process  of  team  collaboration,  with  the  caveat  that  their 
consistency  appears  to  be  domain  dependent.  The  contribution  of  all  five  major  macrocognitive 
processes  during  team  collaboration  was  also  significant.  However,  whether  or  not  this  result  is 
also  dependent  upon  the  problem-solving  domain  is  still  unknown.  The  results  of  the  present  re¬ 
analyses  also  provided  support  for  some  of  the  macrocognitive  subprocesses  in  terms  of  both 
their  consistency  and  contribution.  There  were  also  significant  differences,  depending  on  the 
collaboration  mode  used  by  the  teams,  in  tenns  of  the  collaboration  stages,  macrocognitive  major 
processes,  and  macrocognitive  subprocesses.  Based  on  these  results,  the  next  phase  in  the 
evolution  of  the  conceptual  model  of  team  collaboration  was  proposed.  However,  the  model  will 
no  doubt  continue  to  evolve  since  future  research  is  still  needed  to  further  examine  the 
consistency  and  contribution  across  different  domains  and  in  tenns  of  different  types  of  teams. 
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APPENDIX  A 

THE  NONCOMBATANT  EVACUATION  OPERATION:  RED  CROSS  RESCUE  SCENARIO 

THE  MISSION: 

The  time  is  2:00  am,  January  15th.  Three  American  Red  Cross  workers  are  trapped  in  a  church 
basement  on  a  remote  island  in  the  South  Pacific,  caught  in  the  middle  of  guerilla  warfare.  Your 
mission  is  to  rescue  them  within  24  hr. 

The  situation  is  described  in  the  next  few  pages  along  with  the  assets  of  U.S.  forces  available  to 
rescue  the  workers.  Work  together  to  develop  a  course  of  action,  using  ANY  assets  available  to 
you.  The  detailed  plan  can  be  made  up  of  Army,  Marine,  or  Navy  Seal  assets  (or  a  combination 
of  the  three),  but  must  include  the  following: 

■  a  description  of  how  the  rescuers  will  get  to  the  church 

■  a  description  of  how  they  will  evacuate  the  Red  Cross  workers 

■  and  a  description  of  how  they  will  return  to  the  Army  base  or  aircraft  carrier. 

Choose  the  best  and  most  efficient  solution  to  rescue  the  workers  safely.  Minimize  damage  to  the 
village  and  villagers,  and  avoid  contact  with  the  enemy  if  possible.  Keep  in  mind,  however,  the 
rules  of  engagement  state  that  any  forces  will  defend  themselves  if  needed. 

Good  Luck! 
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BACKGROUND  INFORMATION: 

Drapo  Island 

Date: 


January  15th,  0200  (2:00  AM),  Drapo  Time  Zone 

Location: 

Drapo  Island,  located  750  miles  north  of  Australia,  slightly  northeast  of  the  Solomon  Islands. 

(See  Map  1) 

Size: 


The  island  is  400  miles  wide  by  400  miles  long. 

Topography: 

Drapo  Island  is  made  up  of  volcanic  mountains,  atolls  (ring-like  coral  island  and  reef),  some 
swampy  lowlands,  and  rugged  mountainous  terrain  with  some  low  dense  rainforest.  Because 
much  of  the  island  is  made  up  of  volcanic  mountains,  there  is  a  constant  threat  of  volcanic 
activity.  The  terrain  consists  of  coastal  plains,  swampy  lowlands,  a  large  rainforest  and  mountain 
ranges. 

Location: 

The  church  where  the  workers  are  trapped  is  located  3  miles  from  the  northern-most  shore  of  the 
island.  Directly  in  front  of  the  church  is  a  dirt  road,  which  leads  to  the  ocean  in  one  direction  and 
into  the  main  village  on  the  island  in  the  other  direction.  The  village  is  1.5  miles  from  the 
church.  The  road  also  passes  homes  and  farms  of  the  natives  on  the  way  into  the  town.  The 
church  has  some  vegetation  around  it,  primarily  coconut  trees  and  some  brush,  but  no  heavy 
forest  or  swampland.  The  land  around  it  is  mostly  cultivated  for  farming. 

A  small  port  is  located  on  the  opposite  end  of  the  island  from  the  church,  and  is  heavily  guarded 
by  the  local  military.  To  reach  the  church  from  the  port,  you  must  cross  an  uninhabited  volcanic 
mountain  pass,  which  is  covered  by  dense  rainforest. 

Because  of  the  coral  reefs,  it  is  impossible  to  bring  large  ships  any  closer  than  1  mile  from  the 
coast.  The  shore  is  only  accessible  by  small  boats.  Significant  amounts  of  coral  can  be  seen 
above  the  surface  of  the  water  at  low  tide. 


52 


APPENDIX  A 


NAWCADPAX/TR-2008/43 


There  is  one  paved  highway  around  the  perimeter  of  the  island.  This  road  also  leads  to  the  dirt 
road  where  the  church  is  located.  (See  Maps  2  and  3) 

Climate  and  Weather: 

The  climate  on  Drapo  is  tropical  with  rainy  seasons  from  December  through  March  and  May 
through  October.  There  are  also  periodic  tropical  monsoons.  In  January,  there  are  periods  of 
heavy  fog  in  the  morning,  which  bums  off  as  the  day  progresses.  Visibility  is  usually  clear  by 
noon.  There  are  strong  winds  throughout  the  day. 

Military: 

The  local  military  is  made  up  of  about  100  volunteers  who  support  their  government.  It  has 
recently  been  built  up  in  response  to  increasing  threats  from  rebel  forces.  The  government  has 
limited  military  intelligence  and  limited  analysis  capability.  Drapo  is  in  good  and  peaceful 
standing  with  the  United  States. 

Village: 

The  village  is  home  to  the  government  center,  school,  church,  markets,  and  communication 
center  for  the  island.  Any  weapons  and  military  intelligence  possessed  by  the  island  will  be 
centered  in  the  village.  The  majority  of  the  island’s  homes  are  located  within  1  mile  of  the 
village.  There  are  between  100  and  150  huts  housing  5  to  10  natives  each.  The  village  is  about 
1.5  miles  from  the  church  and  4.5  miles  from  the  northern-most  point  of  the  island. 

The  local  language  is  Draponese.  The  locals  do  NOT  speak  English,  so  if  any  attempt  is  going  to 
be  made  to  communicate  with  the  local  military,  a  translator  will  be  needed. 

Transportation: 

One  main  paved  highway  surrounds  the  perimeter  of  the  island.  (See  Map  2)  Dirt  roads  connect 
the  airport,  homes,  village,  and  the  church  where  Red  Cross  workers  are  trapped. 

There  is  one  main  port,  which  is  used  for  the  export  and  import  of  goods.  Directly  off  the  shore 
of  most  of  the  island  are  atolls,  which  make  it  impossible  for  large  ships  to  come  within  1  mile  of 
the  shore  (with  the  exception  of  the  port). 

There  is  one  main  airport  with  a  paved  runway,  located  200  miles  from  the  village.  The  airport  is 
heavily  guarded  by  the  local  military  to  keep  rebel  forces  out.  The  airport  flies  into  Australia,  the 
nearest  mainland  (750  miles  away),  and  from  there  can  connect  to  other  countries.  (See  Map  3) 
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Communication: 

Communication  on  the  island  and  off  the  island  is  possible,  but  is  limited  by  the  remoteness  of 
the  island.  The  Red  Cross  workers  have  working  mobile  phones;  however,  the  batteries  died 
soon  after  they  contacted  the  American  Red  Cross  Headquarters.  The  church  where  the  workers 
are  trapped  does  not  have  a  telephone.  There  is  limited  radio  communication  on  the  island  (three 
3  AM,  no  FM  stations).  There  are  no  television  stations  on  the  island.  Internet  access  on  the 
island  is  limited  to  government  workers  only. 

Condition  of  Red  Cross  Workers: 

The  workers  are  somewhat  safe  in  one  room  of  the  church.  They  have  no  source  of  water  except 
rainwater.  They  are  able  to  collect  food  easily  from  areas  around  them  but  that  supply  is  limited 
and  venturing  too  far  to  collect  food  is  dangerous.  Rescuers  should  be  aware  that  workers  will 
most  likely  be  dehydrated  and  suffering  from  malnutrition. 

One  of  the  workers  is  a  diabetic  in  need  of  insulin.  He  will  most  likely  go  into  insulin  shock  if 
not  treated  within  24  hr.  Another  worker  has  a  broken  leg.  The  third  worker  appears  to  be 
healthy  at  this  time.  There  is  a  chance  that  workers  could  be  injured  from  the  outside  warfare  and 
that  their  location  may  not  be  safe  for  long.  They  must  be  rescued  within  24  hr. 


54 


APPENDIX  A 


NAWCADPAX/TR-2008/43 


REBEL  FORCES: 

X  The  Rebel  Forces  consists  of  500  trained  soldiers. 

X  They  guard  the  perimeters  of  their  captured  areas  at  ALL  times. 

X  They  have  no  night  vision  capability. 

X  The  have  Stinger  missiles.  (Stinger  missiles  are  hand-held,  infrared,  heat-seeking  rockets 
with  a  range  of  1  -  5  miles.) 

X  Their  small  arms  fire  consists  of  M-16  rifles  (range:  500  yd)  and  9mm  pistols  (range:  25 
yd). 

X  They  have  land  mines.  (It  is  possible  that  local  roads  might  be  mined.) 

X  Their  weapons  storage,  communication  centers,  and  anti-aircraft  locations  are  not  known 
to  the  U.S  or  Drapo  military. 

X  They  have  rocket  propelled  grenades. 

X  Warfare  is  constant  between  rebels  and  local  military. 

X  They  have  easy  access  to  trucks  and  jeeps. 

X  They  are  aware  the  Red  Cross  workers  are  on  the  island,  but  unaware  of  their  exact 
location.  They  are  on-guard  for  possible  rescue  operations. 

NAVY  SEALS  ASSETS: 

Personnel: 

D  7-man  squad  consisting  of: 

~  1  Corpsman 
~  1  Radioman 
~  1  Heavy  Gunners 
~  2  Riflemen  with  M-16  rifles 
~  2  Riflemen  with  M-16  rifles  and  grenade  launchers 
C  Night  vision  capability. 

Q  Need  local  translator  if  communicating  with  villagers. 

D  Ability  to  be  virtually  undetected. 

D  All  Navy  SEALS  are  trained  as  medics. 

C  Navy  SEALS  usually  initiate  covert  operations  from  the  sea. 

C  All  weapons  and  personnel  are  located  on  the  USS  ENTERPRISE. 

C  The  USS  ENTERPRISE  has  full  medical  facilities. 

Transportation  and  Weapons: 


E  Navy  Seahawk  (SH-60):  A  twin-engine 
helicopter  used  for  antisubmarine  warfare, 
drug  interdiction,  cargo  lifts,  and  special 
operations  in  the  day  or  night  regardless  of 
the  weather  conditions. 
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S  Navy  Hornet  (F-18):  A  one  or  two  seat 
supersonic  jet  used  for  air-to-air  (A/ A)  or 
air-to-ground  (A/G)  support. 


C  Zodiac:  A  7-man  inflatable  boat  which  can 
travel  at  speeds  up  to  15  mph. 


ARMY  SPECIAL  FORCES  ASSETS 
Personnel: 

fy.  6-man  Special  Forces  team  consisting  of: 

~  1  Team  Lead 
~  2  Snipers 
~  2  Team  Medics 

~  1  Translator  (Speaks  several  languages,  but  NOT  Draponese.) 
^  Highly  trained  in  multiple  languages. 

^  Night  vision  capabilities. 

^  Weapons:  M- 16,  M-60,  grenades. 

Anny  Special  Forces  usually  initiate  covert  operations  from  the  land. 

Transportation  and  Weapons: 


fr.  Abram  Tanks:  Used  for  enemy 
suppression. 

"Safe 

^  Blackhawk  Helicopter  (UH-60):  A 
twin-engine  helicopter  used  for  special 
operations  in  the  day  or  night, 
regardless  of  the  weather  conditions. 
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AH-64  Apache:  A  twin-engine 
helicopter  used  for  enemy  suppression. 


& 


C-130  Hercules:  A  large  aircraft  used 
for  troop  and  cargo  transport, 
paratrooper  deployment,  and  airborne 
refueling. 


MARINE  ASSETS 
Personnel: 

12-man  team  that  includes: 

~  2  Medical  Corpsmen 
~  Use  of  M- 16,  M-60,  grenades 
~  Night  vision  capabilities 

►  Marine  personnel  are  stationed  on  the  USS  ENTERPRISE. 

Transportation  and  Weapons: 


►  AH-1 :  An  attack  force  helicopter  used  to 
suppress  enemy  troops  and  provide  air 
support. 

’ 

►  CH-53:  A  cargo/transport  helicopter  used 
in  amphibious  and  shore  operations. 

r  t T"  -  '  t 
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MAPS 


The  South  Pacific  Island  of  Drapo 


58 


APPENDIX  A 


NAWCADPAX/TR-2008/43 


Map  1  -  Drapo  Island 
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Map  2  -  Drapo  Island 
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APPENDIX  B 

ENVIRONMENTAL  EXPERT  INFORMATION 

•  Coral  reefs  are  impassable  except  during  high  tide. 

•  Highest  peaks  in  the  mountain  range  reach  20,000  ft,  while  the  lowest  peaks  reach  5,000  ft. 

•  The  water  temperature  is  84°F. 

•  Daytime  temperature  is  90°F. 

•  Night  temperature  is  80°F. 

•  Fog  is  present  on  the  island  from  6  am  to  12  noon. 

•  Sunrise  occurs  at  6  am. 

•  Sunset  occurs  at  7  pm. 

•  While  Drapo  is  home  to  many  indigenous  animals,  no  snakes  can  be  found  on  the  island. 

•  High  tide  is  between  7  and  9  am  and  8  and  10  pm. 

•  On  average,  high  tide  is  about  2  ft. 

•  The  rebels  can  see  12  miles  from  the  beach  (the  distance  of  the  horizon). 

•  The  morning  of  15  January,  there  is  heavy  fog  and  light  drizzle.  The  night  of  16  January, 
there  is  a  full  moon,  cloudy  skies,  and  light  drizzle. 

•  The  waters  around  Drapo  are  a  breeding  ground  for  Great  White  sharks  and  barracuda. 
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APPENDIX  C 

INTELLIGENCE  EXPERT  INFORMATION 

•  It  takes  a  SEAL  team  or  an  Army  Special  Forces  team  45  min  to  reach  the  church  from  the 
shore. 

•  It  takes  a  SEAL  team  or  an  Army  Special  Forces  team  30  min  to  reach  the  church  from  the 
village,  and  30  min  to  return  to  the  village. 

•  Airfield  has  fuel  available  if  ground  refueling  is  needed. 

•  Navy  SEAL  and  Army  Special  Forces  teams  can  parachute  from  a  C-130  (day  or  night). 

•  Church  layout  -  see  below. 

•  Initial  contact  has  been  made  with  local  military. 

•  SEAL  and  Army  Special  Forces  teams  can  use  element  of  surprise. 

•  U.S.  Army  Special  Forces  are  currently  using  a  deserted  anny  base  for  training.  The  base  is 
200  miles  west  of  the  church  where  the  workers  are  trapped. 

•  The  deserted  army  base  where  the  Special  Forces  are  training  and  the  aircraft  carrier  USS 
ENTERPRISE  are  considered  safe  locations. 

•  The  USS  ENTERPRISE  is  anchored  200  miles  east  of  the  church. 

•  The  airport  is  200  miles  from  the  village. 

•  The  local  military  has  three  Toyota  trucks,  bikes,  and  donkeys  available  for  U.S.  force 
transportation. 

Church  Floorplan: 


fflSU 


fro**  ifistcni's 


Hinson  wiratoft 

Upstairs 
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APPENDIX  D 

WEAPONS  EXPERT  INFORMATION 

•  Riflemen  carry  M-16s  (effective  range  500  yd)  and  grenades 

•  Small  arms  fire  can  disable  low-flying  aircrafts 

•  Heavy  gunners  in  SEAL  unit  carry  M-60s  (large  caliber  machine  guns)  with  a  range  of 
1,500  yd. 

•  Suppressive  fire  by  aircraft  can  be  used  to  keep  rebels  from  interfering  with  rescue 
mission,  but  collateral  damage  can  result. 

•  Helicopters  can  be  heard  for  5  miles. 

♦♦♦  U.S.  Navy  or  Marine  F-18  Hornet: 

~  One  or  two  seat  supersonic  jets  that  can  be  used  for  A/A  or  A/G  support 
~  Speed:  Mach  1.7+ 

~  Range:  1,500  miles  (Range  is  defined  as  the  total  miles  that  can  be  flown  without 
refueling.) 

~  Ceiling:  50,000  ft 

~  Weapons:  A/A  missiles,  A/G  smart  bombs,  6,000  rounds  per  minute  cannon 
~  Can  be  flown  in  any  weather,  day  or  night 
~  In-flight  refueling  capabilities 
~  F- 1 8s  are  located  on  the  USS  ENTERPRISE 

♦♦♦  U.S.  Army  Blackhawk  (UH-60)  OR  U.S.  Navy  Seahawk  (SH-60): 

~  Twin  engine  helicopter  used  for  special  ops  in  any  weather,  day  or  night 
~  Speed:  1 84  mph 
~  Range:  395  miles 
~  Ceiling:  14,700  ft 

~  Weapons:  Two  machine  guns  and  missiles  for  A/A  and  A/G  support 
~  Rescue  hoist  cable:  250  ft  with  a  600  lb  lift  capacity 
~  Can  be  refueled  in  flight 
~  Crew:  4 

~  Can  carry  1 1  additional  people 

~  Blackhawks  are  located  on  the  deserted  army  base  where  the  Anny  Special  Forces  are 
conducting  their  exercises 
~  Seahawks  are  based  on  the  USS  ENTERPRISE 

♦♦♦  U.S.  Army  Apache  (AH-64): 

~  Twin  engine  helicopter  used  for  suppression  of  enemy  troops  and  enemy  targets 
~  Speed:  165  mph 
~  Range:  372  miles 
~  Ceiling:  12,500  ft 

~  Weapons:  Missiles  for  A/A  or  A/G  support 
~  Must  be  refueled  on  the  ground 
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~  Crew:  2 

~  Cannot  carry  any  additional  people 
~  Located  on  the  deserted  army  base 

♦♦♦  U.S.  Marine  Cobra  (AH-1): 

~  Attack  force  helicopter  used  to  provide  air  support  (anti-armor,  anti-helicopter,  armed 
escort) 

~  Speed:  170  mph 
~  Range:  300  miles 
~  Ceiling:  12,500  ft 

~  Can  fly  day  or  night,  but  cannot  fly  in  fog 
~  Rescue  hoist  cable:  250  ft  with  a  600  lb  lift  capacity 
~  Must  be  refueled  on  the  ground 
~  Crew:  2 

~  Cannot  carry  any  additional  people 
~  Located  on  the  USS  ENTERPRISE 

❖  U.S.  Marine  CH-53: 

~  Cargo/transport  helicopter  used  in  amphibious  and  shore  operations 
~  Speed:  200  mph 
~  Range:  500  miles 
~  Ceiling:  21,000  ft 
~  Weapons:  None 

~  Rescue  hoist  cable:  250  ft  with  a  600  lb  lift  capacity 
~  Capable  of  refueling  in  flight 
~  Crew:  3 

~  Can  carry  38  additional  people 
~  Cannot  fly  in  fog 

~  Two  inflated  Zodiacs  can  be  air-dropped  out  the  back  ramp 
~  Located  on  the  USS  ENTERPRISE 

❖  U.S.  Army  Hercules  (C- 130): 

~  A  large  aircraft  used  for  troop  and  cargo  transport,  paratrooper  deployment,  and 
airborne  refueling. 

~  Speed:  335  mph 
~  Range:  2,500  miles 
~  Ceiling:  33,000  ft 

~  Weapons:  7.62  mm  mini-guns,  40  mm  cannons,  105  mm  cannon 
~  Crew:  5-9 

~  Can  carry  92  additional  people 

~  Located  on  the  army  base  being  used  for  training  exercises 
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♦♦♦  U.S.  Army  Abram  Tank: 

~  Speed:  45  mph  maximum  (The  Abram  tank  slows  considerable  when  riding  on  a  grade 
or  slope.) 

~  Range:  275  miles 

~  Weapons:  105  mm  rifled  cannon,  .50  caliber  machine  gun,  7.62  mm  swivel  mounted 
machine  gun 
~  Crew:  4 
~  Requires  jet  fuel 

♦♦♦  U.S.  Navy  Zodiac: 

~  A  7-man  inflatable  raft  made  of  durable  rubber  and  polyurethane  fabric 
~  Speed:  15  mph 
~  Weight:  80  lb 

~  Can  be  deflated  and  stowed  in  minutes 
~  Sits  flat  on  top  of  the  water  when  empty 
~  8  in.  displacement  when  loaded  to  full  capacity 
~  It  takes  approximately  1  hr  to  inflate  with  a  hand  pump 

~  Can  be  parachuted  fully  inflated  from  a  height  of  no  more  than  2,000  ft  from  the  back  of 
aCH-53  orC-130. 

~  Zodiacs  are  considered  expendable  items 
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APPENDIX  E 

SCORING  MATRIX  FOR  THE  NEO  SCENARIO 


Out  of  a  possi6Ce  100  points. 


Planning  Card 

Error 

Error 

Type 

Points 

Lost 

Personnel 

Omitting  Personnel  Card 

A 

20 

Transportation 

Omitting  Transportation  Card 

A 

20 

Using  aircraft  that  requires  in  flight  refueling 
without  calculating  refueling  needs 

C 

5 

Calculation  error  rendering  the  solution  impossible 

C 

5 

Using  aircraft  that  requires  in  flight  refueling  (with 
correct  calculations) 

E 

2 

Performing  calculations  incorrectly  with  minimal 
impact 

E 

2 

Using  one  aircraft  that  needs  refueling,  but  others 
that  don't. 

D-E 

2.5 

Weapons 

Omitting  Weapons  Card 

A 

20 

Using  unavailable  weapons 

D 

3 

Failing  to  address  weapons  used  by  selected  military 

D 

3 

Times 

Omitting  Times  Card 

A 

20 

Failing  to  include  required  critical  times  (onset  of 
operation,  contact  with  workers,  evacuation,  return) 

C 

5 

Calculation  error  rendering  the  solution  impossible 

C 

5 

Failing  to  account  for  tides  or  coral  reef  (if  using  sea 
approach) 

D 

3 

Rescuing  workers  during  daylight  hours 

D 

3 

Performing  calculations  incorrectly  with  minimal 
impact 

E 

2 

More  than  0.5  hr  spent  in  church 

E 

2 

Plan 

Omitting  Plan  Card 

A 

20 

Unrealistic  solution  (tanks,  etc.) 

B 

10 

Neglecting  to  include  all  requirements  of  the  plan  as 
listed  in  the  Mission  Statement 

a.  Setting  to  the  church 

b.  Evacuating  the  workers 

c.  Returning  to  the  base  or  ship 

C 

5  for 
each 
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Planning  Card 

Error 

Error 

Type 

Points 

Lost 

Failing  to  address  medical  treatment 

a.  Insulin 

b.  Broken  leg 

C 

2.5  each 

Harming  the  enemy  unnecessarily 

C 

5 

Damaging  the  village  unnecessarily 

c 

5 

Failing  to  arrange  for  translator  if  need  established 

D 

3 

Failing  to  address  detection 

D 

3 

Failing  to  avoid  detection  if  addressed 

C 

2 

Failing  to  avoid  land  mines 

E 

2 

Miscellaneous 

Other  Type  A  error  (e.g.,  omitting  planning  card) 

A 

20 

Other  Type  B  error  (e.g.,  serious  violation  of  mission 
statement,  unrealistic  solution) 

B 

10 

Other  Type  C  error  (e.g.,  moderate  violation  of 
mission  statement,  calculation  error  with  serious 
impact) 

C 

5 

Other  Type  D  error  (e.g.,  minor  violation  of  mission 
statement) 

D 

3 

Other  Type  E  error  (e.g.,  calculation  error  with 
minimal  impact) 

E 

2 

70 


APPENDIX  E 


NAWCADPAX/TR-2008/43 


APPENDIX  F 

EXPERIMENTER  TRANSCRIPT:  NEO  INSTRUCTIONS 
(FACE-TO-FACE  CONDITION) 

In  the  military,  people  often  have  to  work  together  to  solve  problems.  Sometimes  those  people 
are  sitting  around  a  table  in  the  same  room,  but  more  often  than  not,  they  are  scattered  all  over 
the  world.  One  person  may  be  at  the  Pentagon,  one  may  be  in  California,  and  one  may  be  in  a 
cave  in  Afghanistan.  Somehow  all  of  these  people  have  to  work  together  to  solve  whatever 
problem  they  are  working  on.  Right  now,  it  is  usually  done  through  instant  messaging  and  chat. 

We  want  to  help  make  that  communication  more  efficient.  But  before  we  can  do  that,  we  have  to 
understand  how  people  work  together  to  solve  a  problem.  That  is  where  you  come  in.  I  am  going 
to  give  you  a  problem  to  solve  and  I  want  you  to  work  together  as  a  team  to  come  up  with  a 
solution. 

Here  is  the  problem  I  want  you  to  solve:  Three  Red  Cross  workers  are  trapped  in  a  church 
basement  on  a  remote  island  called  Drapo.  They’re  caught  in  a  battle  between  the  guerilla  forces 
(the  bad  guys)  and  the  Draponese  military  (the  good  guys).  There  are  some  U.S.  military  forces 
in  the  area  who  can  help  with  the  rescue. 

Your  job  is  to  come  up  with  a  realistic  plan  to  rescue  the  Red  Cross  workers  using  the  U.S. 
military  forces  in  the  area.  I  will  give  you  all  the  information  you  need  to  solve  the  problem.  You 
do  not  have  to  know  anything  coming  in  to  this.  {Show  the  binders  that  contain  the  NEO 
scenario  and  Expert  Information.}  All  of  the  information  you  need  is  in  these  binders.  Almost  all 
of  the  information  is  identical  in  all  three  binders.  The  last  few  pages,  however,  are  different. 
Each  one  of  you  will  be  assigned  an  area  of  expertise.  One  of  you  will  be  the  Weapons  Expert, 
one  will  be  the  Environmental  Expert,  and  one  will  be  the  Intelligence  Expert.  Again,  all  of  the 
information  you  need  is  in  here. 

I  will  give  you  20  min  to  read  all  the  infonnation,  and  then  60  min  to  work  as  a  team  to  come  up 
with  a  plan.  {Hand  them  the  Final  Plan  form.}  That  plan  has  to  include:  any  U.S.  personnel  who 
will  be  involved;  any  transportation  they  need  to  get  to,  from,  and  around  the  island;  any 
weapons  they  will  need;  critical  times  for  the  mission  (Onset:  when  they  leave  the  base  or  ship  - 
NOT  2  AM  as  listed  in  the  Mission  Statement;  Contact:  when  the  rescuers  come  face  to  face 
with  the  Red  Cross  workers;  Evacuation:  the  time  the  Red  Cross  workers  leave  the  church;  and 
Return  to  Base  or  Ship:  the  time  they  actually  get  back  to  the  base  or  ship);  a  detailed  plan  of  the 
rescue;  and  any  other  comments  you  think  I  should  know. 

At  the  end  of  the  60  min,  I  will  score  your  plan.  While  there  is  no  right  or  wrong  answer,  I  will 
be  scoring  it  based  on  how  realistic  it  is.  Can  this  mission  really  be  carried  out?  We  had  three 
military  experts  work  together  to  come  up  with  their  ideal  plan.  We  cannot  expect  you  to  come 
up  with  the  same  plan  they  came  up  with,  but  when  they  were  developing  their  plan,  they  hit  on 
some  key  points  that,  if  they  are  not  addressed,  no  mission  can  be  successful.  That  is  what  I  will 
be  looking  for  when  I  score  your  plan.  How  realistic  is  this  plan? 
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It  is  important  to  note  that  I  am  only  going  to  score  what  you  have  written  on  your  plan.  I  will  be 
listening  to  your  conversations,  so  I  may  know  what  you  really  meant  to  write  -  but  if  it  is  not 
written  on  your  plan,  I  will  not  score  it.  The  more  detailed  your  plan  is,  the  better  I  will 
understand.  Filling  out  the  plan  is  probably  going  to  take  longer  than  you  think  it  is.  Be  sure  to 
allow  enough  time  to  fill  it  out. 

Again,  I  will  give  you  20  min  to  read  the  binders  to  yourself.  During  that  time,  you  are  welcome 
to  take  any  notes  you  want,  but  you  are  not  allowed  to  talk  to  each  other  at  all  during  that  time. 
After  the  20-min  reading  time,  you  will  have  60  min  to  work  together  to  come  up  with  a  plan. 
You  can  have  your  binders  with  you,  but  please  do  not  hand  your  binder  over  to  your  teammate. 
You  are  more  than  welcome  to  share  whatever  infonnation  is  in  there  -  but  think  of  it  as 
information  in  your  head. 

Before  you  begin,  I  want  to  read  the  Mission  Statement  to  you.  I  apologize  for  reading  out  loud 
to  you,  but  it  is  important  everyone  understands  what  the  goal  is.  {Have  them  follow  along  in 
their  binders  while  you  read.} 

Any  questions? 

Experimenter  after  20  min: 

Remember,  you  have  a  maximum  of  60  min  to  come  up  with  your  Final  Plan.  If  you  finish 
before  that  time,  just  let  me  know. 

Any  questions? 
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APPENDIX  G 

EXAMPLE  OF  A  FINAL  PLAN  FROM  THE  NEO  SCENARIO 


Final  Plan 


_ PERSONNEL 

Xtny  Sea!: 
l'S  Arm)  S  F 
Crew  to  fly  C-130 


TRANSPORTATION 

Stfakni? 

C-130  to  refuel  Seahawi 


WEAPONS 


On  Seahawk: 

2  machine  gun: 

Mt:::le:  fir  air-to-air  and  air-to- 
ground  support 

On  C  -ISO 
2.62  mm  mmi-gun s 
40  mm  cannons 
105  mw  cannon 

Sea!: 

2  rflemen  vt  .'A 1-16  rifle: 

2  rflemen  vt  A (-16  rifle:  and 
grenade  launcher: 

Sight  Vision 


Pass  1  of: 


Grccf 

Dins: 
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CRITIC  AL  TIMES 


1.  Ott'-M  4  j.m 


2.  Ccanct  5:56  am 


i.  Evucuicioa:  5:45 


4.  Rtrara  co  Ba-.r  or  Ship:  6:54 


DETAILED  PLAN 


4  am  leone  USS  and  fly  to  church. 
Ha\t  SEAL  Team  fit::  rope  down. 
Ho;;:  rid  cros;  and  team  aboard.  Fiy 
outtoward  Army  base  Ha\e  C-liO 
mu:  4  6:30  for  refuel  Land  on 
Arm y  ban  %  6:54. 

Local  military  oj  to  beep  rebel;  busy 
and  out  of  area  if  possible 


ADDITIONAL  COMMENTS 

Xfflaman  and  night  vision  to  hold  off 
any  rebel;  vho  may  get  through 

Flight  i a  night  to  a\otd  detection 

SEAL  S  -  Smallest  *  of  all  medic; 

Medical  attention  a;  soon  a;  hoist  i: 
complete 

Can  defend  themsehe;  w  weapon; 


Pin  2  of  2 
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APPENDIX  H 

NEO  SCENARIO  INFORMATION  UPDATE:  DYNAMIC  KNOWLEDGE  CONDITION 


NEWS  ALERT  !!!!! 

1.  Intelligence  reports  indicate  that  the  enemy  forces  have  taken 
control  of  the  village.  Enemy  strongholds  now  extend  from  the 
northern  tip  of  the  island,  southward  for  75  miles. 

2.  The  fog  encompassing  the  island  will  lift  at  0900  (9  am). 

3.  Enemy  patrol  boats  have  been  sighted  off  the  eastern-most  point 
of  the  island.  While  the  patrol  boats  can  travel  at  speeds  up  to 
30  mph,  they  appear  to  be  staying  along  the  shore,  within  15  miles 
to  the  north  and  south  of  the  eastern  tip. 


75 


APPENDIX  H 


NAWCADPAX/TR-2008/43 


APPENDIX  I 

IDEAL  SOLUTION  TO  THE  NEO  SCENARIO 


Solution: 

>  Key  Points: 

•  Covert  operation 

•  Conducted  at  night,  during  high  tide 

•  Use  Navy  SEALs  and  Marine  CH-53  (does  not  need  refueling) 

•  Weapons:  M-16  rifles  and  grenade  launchers 

2.  1800  hr:  SEAL  team  leaves  USS  ENTERPRISE  on  a  CH-53. 

3.  1900  hr:  SEAL  team  drops  from  a  height  of  500  ft  into  the  ocean  with  two  inflated 
Zodiacs.  The  drop  is  12  miles  off  the  shore  (over  the  horizon.  The  CH-53  returns  to  the 
base  or  USS  ENTERPRISE  for  refueling. 

4.  2000  hr:  SEAL  team  reaches  the  shore  and  hides  the  Zodiacs.  Proceeds  to  the  church, 
being  careful  to  stay  off  the  roads  because  of  possible  mines. 

5.  2045  hr:  Reaches  church,  makes  contact  with  workers. 

6.  2100  hr:  Leaves  church,  carrying  wounded  and  providing  medical  care  along  the  way. 

7.  2145  hr:  Reaches  shore.  Uses  the  two  hidden  Zodiacs  to  motor  out  12  miles. 

8.  2245  hr:  Rendezvous  with  CH-53  (altitude:  no  higher  than  250  ft).  Airlifts  workers  onto 
the  helicopter.  SEAL  team  sinks  Zodiacs. 

9.  2345  hr:  Return  to  USS  ENTERPRISE  for  additional  medical  treatment. 
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APPENDIX  J 

DEFINITIONS  AND  EXAMPLES  FOR  THE  MACROCOGNITIVE  MAJOR  PROCESSES 

A. )  Individual  Knowledge  Building:  Individual  team  members  ask  for  clarification  of  data  or 

infonnation,  or  respond  to  clarification  requested  by  other  team  members. 

o  “I  am  the  environmental  expert.”  (if  someone  asked  who  had  infonnation  about  the  weather ) 

o  “Where  do  we  write  up  our  final  plan?” 

o  “We  have  30  min  remaining.”  (if  someone  asked  how  much  time  was  left  to  do  this) 
o  “Don’t  they  have  a  translator?” 
o  “I  don’t  think  anyone  has  a  translator  for  Draponize.” 

B. )  Team  Knowledge  Building:  All  team  members  participate  in  clarifying  information  (e.g., 

answering  a  question)  to  build  team  knowledge.  (NOT  solution  options.) 

o  “Are  there  windows  in  the  church?”  (E) 
o  “Eight  windows  in  the  church.”  (W) 
o  “Eight  windows.”  (I) 

o  “The  only  problem  is  they  are  not  going  to  do  us  any  good  because  they  (workers)  are  in  an  inside 
room.”  (I) 

o  “Let’s  see.  The  fog  starts  at  6?”  (W) 
o  “Yes,  it  does,  and  it’ll  last  until  noon.”  (E) 
o  “And  you  had  said  there’s  heavy  fog  and  light  drizzle  too.”  (I) 
o  “Yeah,  all  day.”  (E) 

C. )  Developing  Shared  Problem  Conceptualization:  Team  members  sharing  their 

understanding  of  problem  goals,  characteristics  of  the  environment  and  rules  for  operating 
the  generation  of  quality  problem  solutions.  (Unlike  “B”,  does  NOT  need  to  include  all 
team  members.  NOT  solution  options.  NOT  an  answer  to  a  question.) 

Example:  Goals 

o  “Okay.  So  we  have  24  hr  to  rescue  the  Red  Cross  workers.” 
o  “Yes,  and  we  have  about  50  min  to  come  up  with  a  feasible  plan  to  do  it.” 

Example:  Characteristics  of  Environment 

o  We  can  only  get  small  boats  in  at  high  tide,  which  is  7-9  am  and  8-10  pm. 

Example:  Rules  -  Quality  Problem  Solutions 

o  “Our  goal  is  to  rescue  the  Red  Cross  workers  and  get  in  and  out  undetected  using  either  the  army, 
navy,  or  the  marines.” 

D. )  Team  Consensus  Development:  Raisins  (see  example  1.)  and/or  discussins  solution 

options  (see  example  2)  (not  necessary  to  include  all  three  team  members)  or  Collective 
Agreement  (see  examples  3  and  4)  by  team  members  on  a  particular  final  solution  option 
(i.e.,  each  team  member  does  not  have  to  agree  on  the  solution  option  but  as  a  team  they 
need  to  agree  on  the  final  solution  option).  (For  final  solution  option  only;  Must  include 
all  three  team  members,  and  must  end  in  final  consensus/agreement.) 

Example  1:  Raising  an  Option 
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o  “I’m  thinking  the  Navy  SEALS  and  the  Marines.  They  can  both  see  at  night  and  the  enemy  can’t. 
That’s  a  big  advantage  for  getting  in  and  out  undetected.  Besides,  they  work  well  together.” 

Example  2:  Discussing  Solution  Options 

o  I  think  we  should  use  an  1 1  man  team:  7  SEAL  and  4  Marines, 
o  Why  only  marines  and  seals? 
o  What  do  you  guys  think? 
o  Because  Navy  and  Marines  work  hand  and  hand. 

Example  3:  Collective  Agreement 
o  “I  agree  with  the  final  plan.”  (W) 

o  “I’ll  go  along  with  what  we  have.  Although  I’m  still  not  sure  why  we  need  the  Marines  as  well  as 
the  Navy  SEALS.”  (I) 
o  “I’m  okay  with  our  plan.”  (E) 

Example  4:  Collective  Agreement 

o  “I’m  thinking  that  we  use  the  Seahawk.  It’s  used  for  special  operations  and  can  go  395  miles 
without  refueling.”  (W) 

o  “Sounds  good  if  it  can  fly  in  the  fog  and  drizzle.”  (E) 
o  It  can  fly  day  or  night,  regardless  of  the  weather  conditions.”  (W) 
o  “That  sounds  good  to  me.”  (E) 
o  “Yeah,  sounds  like  a  good  plan  to  me  too.”  (I) 
o  “Okay.  Seahawk  it  is.”  (W) 

E.)  Outcome  Appraisal:  Team  evaluation  (all  team  members)  of  selected  solution  option 
against  problem  solving  goal  Team  revises  solution  option  if  option  does  not  meet  goal. 

o  “Our  plan  to  use  the  Navy  SEALS,  Blackhawk,  and  the  Zodiak  meets  all  the  main  criteria  for 
rescuing  the  Red  Cross  workers.”  (I) 
o  “Agreed.”  (E) 
o  “Okay.”  (W) 

o  “The  Navy  SEALS  using  the  Blackhawk  and  the  Zodiak  can  accomplish  the  mission,  but  our  plan 
cannot  be  completed  within  24  hr.”  (I) 

o  “We  need  to  relook  at  the  details  of  the  plan.  How  about  if  we  leave  earlier?”  (W) 
o  “That  might  work.”  (E) 
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APPENDIX  K 

DEFINITIONS  AND  EXAMPLES  FOR  THE  MACROCOGNITIVE  SUBPROCESSES 

Macrocognitive  Subprocesses: 

♦  1.)  Iterative  Information  Collection:  Collecting  and  analyzing  information  to  come  up 
with  a  solution  but  no  specific  solution  mentioned. 

o  “I  noticed  the  rebels  don’t  have  night  vision  and  it’s  foggy  in  the  morning  till  10  am  ( data );  that’s 
to  our  advantage.”  ( information  —  no  mention  of  specific  use  of  this  information  towards  the 
solution) 

o  “It  seems  we  have  better  NVG  capability  than  the  rebels”  (providing  analyzed  information) 
o  “1  think  we  should  first  start  talking  about  our  expertise.”  (i.e.,  approach  to  develop  a  solution) 
o  “They  have  medical  training.”  ( referring  to  the  SEALS) 

♦  2.)  Individual  Task  Knowledge  Development:  An  individual  team  member  asking  for 
clarification  to  data  or  information  about  the  task;  or  response  to  clarification  about  the  task 
( Course  of  Action) . 


o  “Where  do  we  write  up  our  final  plan?” 
o  “I’m  going  to  talk  and  you  write  that  in  the  final  plan.” 
o  “Do  you  want  to  do  a  quick  game  plan  first?” 

♦  3.)  Individual  Mental  Model  Development:  Individual  team  member  using  available 
information  to  increase  his/her  knowledge  representation  of  the  problem  situation. 

o  “It’s  two  in  the  morning  in  January  and  January  is  the  rainy  season  for  this  island;  the  island  is 
750  miles  north  of  Australia  and  is  mostly  rain  forest  and  vegetation;  can  only  get  battleships 
within  a  mile  of  coral  and  at  low  tide  can’t  get  a  single  man  raft  onto  the  island;  before  noon 
heavy  fog  and  by  noon  it  will  be  gone.” 

♦  4.)  Team  Pattern  Recognition  and  Trend  Analysis:  The  team  as  a  whole  identifies  a 
pattern  of  data,  infonnation  or  knowledge. 

o  “It  seems  like  all  the  needed  assets  (SEALS,  Blackhawk,  F-18’s)  are  located  on  the  USS 
ENTERPRISE.”(E)  (needed  assets  are  the  patterned  information) 
o  “Yes.”  (I) 
o  “Yes.”  (W) 

♦  5.)  Team  Mental  Model  Development:  The  increasing  similarity  between  an  individual’s 
knowledge  representation  and  the  team’s  knowledge  representation  through  the  process  of 
individual  team  members  convincing  other  team  members  to  accept  specific  data, 
information  or  knowledge. 

o  “I’m  not  sure  about  the  Navy  SEALS  for  this  operation.”  (E) 

o  “The  SEALS  still  sound  good  to  me.  Plus  they  are  emergency  medically  trained  and  we  need 
that.”  (I) 
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o  “Now  that  you  pointed  that  out  to  me  (about  the  medic).  I  see  what  you  mean  about  using  the 
SEALS.”  (E) 

o  “Let’s  go  with  the  SEALS  then.”  (W) 

♦  6.)  Recognition  of  Expertise:  When  an  individual  recognizes/acknowledges  another 
individual  as  an  expert  and  accepts  specific  data,  information,  or  knowledge  from  that 
individual  as  true. 

o  “We  need  to  arrive  on  the  island  by  small  boat  during  high  tide  due  to  the  coral  reefs.”  (E) 
o  “You’re  the  environmental  expert.  So,  using  the  zodiak,  then.  Just  tell  me  what  time  we  need  to 
arrive  and  we’ll  go  with  that.”  (W) 
o  “So,  where  do  you  want  to  enter  from?”  (E) 
o  “That’s  her  expertise.”  (W) 
o  “They  enter  from  this  part... the  stairs.”  (I) 
o  “From  the  front  or  the  side?”  (E) 
o  “The  front.”  (I) 
o  “So — by  the  stairs  then.”  (W) 
o  “Yeah.”  (I) 

♦  7.)  Sharing  Unique  Knowledge:  An  exchange  process  where  any  information  uniquely 
held  by  an  individual  (e.g.,  weapons  information,  intelligence  information)  is  made  available 
to  all  other  group  members  and  the  group  uses  this  infonnation  in  their  option  selection 
process. 


o  “I  have  information  on  the  sunrise/sunset  times:  7  am/7  pm.  (E) 
o  “We’ll  leave  after  7  pm  then.  Since  it’ll  be  dark.”  (W) 

♦  8.)  Uncertainty  Resolution:  The  progressive  minimization  of  sources  of  uncertainty  in  a 
decision  environment. 

o  “And  they  have — the  local  military  has  three  Toyota  trucks,  bikes,  and  donkeys  available  for  us  to 
use.” 

o  “Trucks  are  faster,  but  they  would  be  able  to  be  heard.” 
o  “The  donkeys,  you  never  know  if  they’re  going  to  cooperate  or  not.” 
o  “So,  if  you’re  going  to  rely  on  one  of  the  three,  I  would  think  the  bikes  then.” 
o  “Can  it  move?” 
o  “What?  The  forces?” 
o  “No,  the  Enterprise.” 
o  “Oh,  yeah.  I  assume  so.” 
o  “It’s  not  a  stationary  boat?” 

o  “I  don’t  think  so.  It  doesn’t  really  give  that  much  information  on  it.” 
o  “I  was  under  the  impression  that  you  couldn’t  move  it.” 
o  “Yeah,  me  too.” 

o  “Okay.  Then  I  guess  we’re  going  to  have  to  take  something  smaller  from  the 
Enterprise  and  use  that  to  get  onto  the  island.” 

♦  9.)  Knowledge  Interoperability:  The  process  of  individual  team  member’s  exchanging 
their  knowledge  of  the  problem  situation  such  that  agreement  is  reached  among  team 
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members  with  respect  to  a  common  understanding  of  the  topic’s  meaning  (Implied  agreement 
is  okay  as  long  as  all  three  team  members  are  involved  in  the  conversation.) 

o  “So  the  rebel  forces  don’t  know  where  the  Red  Cross  people  are?” 
o  “Right.” 

o  “They  have  no  idea  that  they  are  in  the  church.” 
o  “So,  okay.  That’s  good.” 

o  “The  problem  is  getting  to  the  church  and  getting  them  out  of  the  church  and  maybe  back  to  the 
village  I  would  think.” 

o  “Yeah.” 

o  “Since  once  you’re  in  the  village  you're  ok.” 

o  “Right,  so  let’s  figure  out  how  to  get  them  from  the  church  to  the  village.” 

♦  10.)  Visualization  and  Representation  of  Meaning:  Visualization  is  where  individual 
team  members  use  methods  such  as  graphs  and  pictures  to  transfer  meaning  to  other  team 
members.  Representation  is  where  individual  team  members  use  methods  such  as  note  pads 
to  sort  data  and  information  into  meaningful  chunks. 

o  “Let’s  use  our  note  pads  to  sort  the  information  we  have  into  categories.” 
o  “Let’s  draw  out  our  basic  timeline  so  far.” 

♦  11.)  Knowledge  Sharing:  Pieces  of  information  passed  to  another  team  member. 

o  “I  am  the  weapons  expert.” 
o  “We  only  have  10  min  left.” 
o  “I  need  to  know  who  the  intelligence  person  is.” 
o  “Okay,  we’re  ready  to  begin.” 

♦  12.)  Knowledge  Transfer:  The  act  of  exchanging  useful,  actionable  knowledge  among 
team  members.  The  term  knowledge  is  defined  and  used  according  to  the  definition  and 
principles  of  Bellinger, et  al,  2004.  Knowledge  represents  a  pattern  that  connects  and 
generally  provides  a  high  level  of  predictability  as  what  is  described  or  what  will  happen 
next. 

o  “If  the  water  level  is  too  low,  the  sharp  reefs  will  cut  the  bottom  of  the  boat.  Therefore  we  must 
drop  the  Zodiak  during  high  tide  so  that  the  SEALS  can  reach  the  shore  without  the  reefs  causing 
damage  to  the  boat.” 

o  “We  got  to  keep  in  mind  that  if  we  use  that,  they  got  to  refuel  at  the  airport.” 
o  “Yeah,  refuel.  Okay.” 

♦  13.)  Team  Shared  Understanding:  Discussion  among  all  team  members  on  a  particular 
topic  or  data  item  (i.e.,  discussion  does  not  involve  answering  questions). 

o  “No.  We  don’t  have  a  translator  that  speaks  Draponize.”  (I) 
o  “But  it  says  here  we  have  a  translator.”  (W) 
o  “He  is  highly  skilled  in  many  different  languages.”  (I) 

o  “Then  what  you  are  hoping  is  that  there  is  a  translator  on  the  island  that  speaks  Draponize  and 
another  language  that  our  translator  also  speaks.”  (E) 
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♦  14.)  Critical  Thinking:  Critical  thinking  impacts  decision  processes  and  outcomes  and  can 
be  indirectly  measured  through  these:  issues  considered  in  an  assessment  of  the  plan, 
conflicting  pieces  of  evidence  identified,  explanations  of  conflict  generation. 

o  “We  need  to  figure  out  the  critical  times  when  we  can  leave  and  at  what  time  we  would  return  by. 

So  each  of  us  needs  to  provide  all  the  information  we  have  about  time  constraints.” 
o  “First,  we  need  to  know  which  branch  of  the  military  has  a  translator  that  speaks  Draponize  if 
we’re  going  to  interact  with  the  villagers.”  (W) 
o  “None  of  the  branches  have  a  translator  that  speaks  Draponize.”  (I) 
o  “Okay,  that  could  be  a  problem.  We  may  not  be  able  to  use  the  villagers.”  (W) 
o  “We  need  to  arrive  in  the  middle  night  so  that  the  rebels  can’t  see  us.” 
o  “We  can  only  get  on  the  island  by  boat  during  high  tide.” 

♦  15.)  Intuitive  Decision  Making:  Team  members  recognizing  the  situation  as  typical,  so 
they  immediately  know  what  course  of  action  they  will  do.  They  immediately  know  the 
goals,  priorities,  and  the  steps  of  the  course  of  action  in  the  given  situation. 

o  “I’ve  seen  movies  where  they  do  rescues  kind  of  like  this,  and  in  them  the  SEALS  and  Marines 
usually  work  really  well  together.  Usually,  you  see  them  with  boats  and  helicopters,  and  we  have 
use  of  both.” 

o  “Yeah,  I’ve  seen  movies  like  that  too.  SEALS  do  the  rescue,  Marines  are  there  to  watch  their 
backs  and  fight  off  the  enemy  on  the  way  in  and  out.  Let’s  plan  on  doing  our  rescue  the  same 
way.” 

♦  16.)  Solution  Option  Generation:  Generating  reasonable  alternatives  in  a  decision 
problem  that  satisfy  the  list  of  requirements. 

o  “Let’s  see,  we  need  to  get  our  guys  on  the  island  to  rescue  the  workers.  Since  the  coral  reefs  are  a 
problem,  two  possible  ways  around  it  to  get  us  onto  the  island  are  using  a  helicopter  to  drop 
rescuers  right  onto  the  island,  or  to  use  the  helicopter  to  drop  the  Zodiak  during  high  tide.” 
o  “High  tide  let’s  us  avoid  the  reef  from  7-9  am;  also  there  is  heavy  fog  then  with  light  drizzle.  I’d 
recommend  us  going  in  by  zodiac  during  that  time  frame.” 
o  “We  have  a  few  options  to  do  this.  Should  we  parachute  in  or  take  one  of  the  navy  SEAL  type 
zodiac  boats?  Also,  should  we  go  through  the  shorter  more  dangerous  enemy  territory  to  save 
time,  or  should  we  stay  inside  the  blue  areas  to  get  to  the  Red  Cross  workers?” 

♦  17.)  Team  Negotiation  of  Solution  Alternatives:  Changes  of  solution  alternatives  by  the 
total  team  to  reach  agreement. 

Weapons  Example: 

•  “So  now  we’re  only  using  the  F-18  for  backup  cover.”  (E) 

•  “F-18  is  just  on  standby.”  (W) 

•  “Yes,  that’s  the  new  plan.”  (E) 

•  “Used  to  take  out  oncoming  enemy  threat.”  (I) 
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♦  18.)  Feedback  Interpretation:  The  whole  team  discusses  the  selected  solution  option 
against  meeting  the  problem  goal  resulting  in  either  completely  meeting  the  goal  or  areas  that 
need  to  be  discussed  further. 

o  “Our  plan  meets  the  24  hr  rescue  requirement  with  minimum  enemy  contact.”  (I) 
o  “Agreed.”  (W) 
o  “Yes”  (E) 

♦  19.)  Replanning:  Changes  made  to  the  initial  plan  until  the  final  plan  is  achieved.  Note: 
initial  plan  is  defined  as  the  preliminary  course  of  action  developed  by  the  team.  All  behavior 
leading  up  to  the  initial  plan  is  not  included  in  the  replanning  measure. 

o  Okay,  so  our  rescue  plan  doesn’t  meet  the  24  hr  guideline.  We’ll  just  make  a  few  changes.” 
o  “Since  we  need  to  be  back  within  24  hr,  let’s  just  leave  earlier. .  .just  so  long  as  it’s  still  dark  out.” 

Macrocognitive  Secondary  Measures: 

♦  20.)  Building  Common  Ground:  Common  ground  equals  the  amount  of  redundant  terms 
(x)  emerging  within  the  group  activity  over  the  total  number  of  words  (n)  generated  by  the 
group  (i.e.,  CG  =  x/n;  whereas  the  lower  the  number  the  greater  the  common  ground  among 
the  team).  Or  BCG  =  (#4  +  #5  +  #6  +  #8  +  #9  +  #13  +  #l  7)/(Total  #  Utterances). 

♦  21.)  Mental  Simulation:  Individual  team  members  or  the  whole  team  using  their  mental 
models  (i.e.,  individual  or  team,  respectively)  to  run  a  mental  simulation  of  what  might 
happen  over  time.  Pattern  matching  between  features  of  the  current  situation  and  those  of 
previous  similar  situations  in  order  to  base  projections  on  the  outcomes  of  the  current 
situations.  Measures  of  mental  simulation  content  could  include  pre/post  session 
questionnaires,  and  concept  maps;  mental  model  occurrence  measures  could  include 
detection  of  communication  utterances  describing  comparisons  to  similar  problems  and 
projections  about  current  problem.  Or  MS  =  Out  of  all  #  3  and  #  5,  see  which  ones  have 
“...what  if....”  type  statements.  Tag  these  as  MS  then  total  these  to  come  up  with  MS 
score  and  divide  by  the  total  #  of  Utterances. 

♦  22.)  Storyboarding;  The  process  of  visual  thinking  and  planning,  which  allows  a  team  to 
brainstorm  together.  In  addition,  the  team  can  place  and  arrange  their  ideas  on  cards  before 
taking  action.  Measures  for  effective  storyboarding  include  counting  the  number  of 
cards/utterances  necessary  for  explaining  a  concept  or  idea.  Or  SB  =  (#  1  +  #  3  +  #  7+  #  12  + 

#  16  +  #  19)/(Total  #  Utterances). 
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